2023-04-25 09:53:31,206 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f 2023-04-25 09:53:31,224 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-25 09:53:31,243 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-25 09:53:31,244 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/cluster_2925ad82-f88f-75b8-db74-5e3e3d9f1396, deleteOnExit=true 2023-04-25 09:53:31,244 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-25 09:53:31,245 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/test.cache.data in system properties and HBase conf 2023-04-25 09:53:31,246 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/hadoop.tmp.dir in system properties and HBase conf 2023-04-25 09:53:31,246 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/hadoop.log.dir in system properties and HBase conf 2023-04-25 09:53:31,247 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-25 09:53:31,247 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-25 09:53:31,247 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-25 09:53:31,368 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-25 09:53:31,775 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-25 09:53:31,780 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-25 09:53:31,781 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-25 09:53:31,782 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-25 09:53:31,782 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-25 09:53:31,783 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-25 09:53:31,783 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-25 09:53:31,783 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-25 09:53:31,784 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-25 09:53:31,784 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-25 09:53:31,785 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/nfs.dump.dir in system properties and HBase conf 2023-04-25 09:53:31,785 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/java.io.tmpdir in system properties and HBase conf 2023-04-25 09:53:31,785 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-25 09:53:31,786 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-25 09:53:31,786 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-25 09:53:32,391 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-25 09:53:32,395 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-25 09:53:32,691 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-25 09:53:32,879 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-25 09:53:32,907 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-25 09:53:32,950 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-25 09:53:32,982 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/java.io.tmpdir/Jetty_localhost_35325_hdfs____s23y66/webapp 2023-04-25 09:53:33,129 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35325 2023-04-25 09:53:33,145 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-25 09:53:33,146 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-25 09:53:33,703 WARN [Listener at localhost/35439] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-25 09:53:33,785 WARN [Listener at localhost/35439] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-25 09:53:33,805 WARN [Listener at localhost/35439] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-25 09:53:33,812 INFO [Listener at localhost/35439] log.Slf4jLog(67): jetty-6.1.26 2023-04-25 09:53:33,821 INFO [Listener at localhost/35439] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/java.io.tmpdir/Jetty_localhost_40907_datanode____.9j347p/webapp 2023-04-25 09:53:33,921 INFO [Listener at localhost/35439] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40907 2023-04-25 09:53:34,327 WARN [Listener at localhost/33705] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-25 09:53:34,889 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x328ae8297063c7d6: Processing first storage report for DS-73b55fbd-4993-4118-898a-5da5bdc68e53 from datanode 54203c7c-a588-4776-8ed2-7bd6392ca367 2023-04-25 09:53:34,891 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x328ae8297063c7d6: from storage DS-73b55fbd-4993-4118-898a-5da5bdc68e53 node DatanodeRegistration(127.0.0.1:36589, datanodeUuid=54203c7c-a588-4776-8ed2-7bd6392ca367, infoPort=35675, infoSecurePort=0, ipcPort=33705, storageInfo=lv=-57;cid=testClusterID;nsid=1685660382;c=1682416412467), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-25 09:53:34,892 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x328ae8297063c7d6: Processing first storage report for DS-203fc285-0186-4b4b-9830-53bf85a9e013 from datanode 54203c7c-a588-4776-8ed2-7bd6392ca367 2023-04-25 09:53:34,892 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x328ae8297063c7d6: from storage DS-203fc285-0186-4b4b-9830-53bf85a9e013 node DatanodeRegistration(127.0.0.1:36589, datanodeUuid=54203c7c-a588-4776-8ed2-7bd6392ca367, infoPort=35675, infoSecurePort=0, ipcPort=33705, storageInfo=lv=-57;cid=testClusterID;nsid=1685660382;c=1682416412467), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-25 09:53:34,971 DEBUG [Listener at localhost/33705] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f 2023-04-25 09:53:35,042 INFO [Listener at localhost/33705] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/cluster_2925ad82-f88f-75b8-db74-5e3e3d9f1396/zookeeper_0, clientPort=60014, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/cluster_2925ad82-f88f-75b8-db74-5e3e3d9f1396/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/cluster_2925ad82-f88f-75b8-db74-5e3e3d9f1396/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-25 09:53:35,057 INFO [Listener at localhost/33705] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60014 2023-04-25 09:53:35,064 INFO [Listener at localhost/33705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:35,066 INFO [Listener at localhost/33705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:35,729 INFO [Listener at localhost/33705] util.FSUtils(471): Created version file at hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c with version=8 2023-04-25 09:53:35,729 INFO [Listener at localhost/33705] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/hbase-staging 2023-04-25 09:53:36,057 INFO [Listener at localhost/33705] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-25 09:53:36,571 INFO [Listener at localhost/33705] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-25 09:53:36,602 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 09:53:36,603 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-25 09:53:36,603 INFO [Listener at localhost/33705] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-25 09:53:36,603 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 09:53:36,604 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-25 09:53:36,754 INFO [Listener at localhost/33705] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-25 09:53:36,835 DEBUG [Listener at localhost/33705] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-25 09:53:36,931 INFO [Listener at localhost/33705] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33039 2023-04-25 09:53:36,942 INFO [Listener at localhost/33705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:36,945 INFO [Listener at localhost/33705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:36,966 INFO [Listener at localhost/33705] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33039 connecting to ZooKeeper ensemble=127.0.0.1:60014 2023-04-25 09:53:37,007 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:330390x0, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-25 09:53:37,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:33039-0x101babd1af30000 connected 2023-04-25 09:53:37,036 DEBUG [Listener at localhost/33705] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 09:53:37,037 DEBUG [Listener at localhost/33705] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 09:53:37,040 DEBUG [Listener at localhost/33705] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-25 09:53:37,048 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33039 2023-04-25 09:53:37,048 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33039 2023-04-25 09:53:37,049 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33039 2023-04-25 09:53:37,049 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33039 2023-04-25 09:53:37,050 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33039 2023-04-25 09:53:37,056 INFO [Listener at localhost/33705] master.HMaster(444): hbase.rootdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c, hbase.cluster.distributed=false 2023-04-25 09:53:37,124 INFO [Listener at localhost/33705] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-25 09:53:37,125 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 09:53:37,125 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-25 09:53:37,125 INFO [Listener at localhost/33705] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-25 09:53:37,125 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-25 09:53:37,125 INFO [Listener at localhost/33705] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-25 09:53:37,130 INFO [Listener at localhost/33705] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-25 09:53:37,133 INFO [Listener at localhost/33705] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39407 2023-04-25 09:53:37,135 INFO [Listener at localhost/33705] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-25 09:53:37,141 DEBUG [Listener at localhost/33705] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-25 09:53:37,142 INFO [Listener at localhost/33705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:37,145 INFO [Listener at localhost/33705] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:37,147 INFO [Listener at localhost/33705] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39407 connecting to ZooKeeper ensemble=127.0.0.1:60014 2023-04-25 09:53:37,151 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:394070x0, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-25 09:53:37,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39407-0x101babd1af30001 connected 2023-04-25 09:53:37,152 DEBUG [Listener at localhost/33705] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 09:53:37,153 DEBUG [Listener at localhost/33705] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 09:53:37,154 DEBUG [Listener at localhost/33705] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-25 09:53:37,155 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39407 2023-04-25 09:53:37,155 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39407 2023-04-25 09:53:37,155 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39407 2023-04-25 09:53:37,160 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39407 2023-04-25 09:53:37,160 DEBUG [Listener at localhost/33705] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39407 2023-04-25 09:53:37,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:37,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-25 09:53:37,175 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:37,198 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-25 09:53:37,198 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-25 09:53:37,199 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:37,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-25 09:53:37,201 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,33039,1682416415885 from backup master directory 2023-04-25 09:53:37,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-25 09:53:37,206 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:37,206 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-25 09:53:37,207 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-25 09:53:37,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:37,210 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-25 09:53:37,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-25 09:53:37,703 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/hbase.id with ID: 5dab8261-6131-4928-9dfe-dddf5a2ff949 2023-04-25 09:53:37,748 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-25 09:53:37,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:37,804 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2881fe27 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:37,837 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bfb9b11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:37,861 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:53:37,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-25 09:53:37,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-25 09:53:37,880 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-25 09:53:37,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:37,886 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:37,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 09:53:37,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store-tmp 2023-04-25 09:53:37,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:37,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-25 09:53:37,958 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:53:37,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:53:37,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-25 09:53:37,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:53:37,958 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:53:37,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 09:53:37,960 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/WALs/jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:37,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33039%2C1682416415885, suffix=, logDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/WALs/jenkins-hbase4.apache.org,33039,1682416415885, archiveDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/oldWALs, maxLogs=10 2023-04-25 09:53:38,045 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36589,DS-73b55fbd-4993-4118-898a-5da5bdc68e53,DISK] 2023-04-25 09:53:38,056 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:38,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/WALs/jenkins-hbase4.apache.org,33039,1682416415885/jenkins-hbase4.apache.org%2C33039%2C1682416415885.1682416417994 2023-04-25 09:53:38,127 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36589,DS-73b55fbd-4993-4118-898a-5da5bdc68e53,DISK]] 2023-04-25 09:53:38,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:53:38,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:38,133 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 09:53:38,134 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 09:53:38,213 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-25 09:53:38,222 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-25 09:53:38,253 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-25 09:53:38,267 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:38,274 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-25 09:53:38,276 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-25 09:53:38,291 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-25 09:53:38,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:53:38,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71187709, jitterRate=0.06077952682971954}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-25 09:53:38,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 09:53:38,297 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-25 09:53:38,317 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-25 09:53:38,317 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-25 09:53:38,319 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-25 09:53:38,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-25 09:53:38,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-25 09:53:38,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-25 09:53:38,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-25 09:53:38,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-25 09:53:38,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-25 09:53:38,412 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-25 09:53:38,414 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-25 09:53:38,420 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-25 09:53:38,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-25 09:53:38,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:38,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-25 09:53:38,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-25 09:53:38,445 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-25 09:53:38,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-25 09:53:38,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-25 09:53:38,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:38,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,33039,1682416415885, sessionid=0x101babd1af30000, setting cluster-up flag (Was=false) 2023-04-25 09:53:38,466 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:38,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-25 09:53:38,478 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:38,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:38,489 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-25 09:53:38,490 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:38,493 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.hbase-snapshot/.tmp 2023-04-25 09:53:38,566 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(952): ClusterId : 5dab8261-6131-4928-9dfe-dddf5a2ff949 2023-04-25 09:53:38,571 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-25 09:53:38,578 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-25 09:53:38,578 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-25 09:53:38,582 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-25 09:53:38,583 DEBUG [RS:0;jenkins-hbase4:39407] zookeeper.ReadOnlyZKClient(139): Connect 0x42bc3cbc to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:38,588 DEBUG [RS:0;jenkins-hbase4:39407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c7bda9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:38,589 DEBUG [RS:0;jenkins-hbase4:39407] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@484457cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-25 09:53:38,614 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39407 2023-04-25 09:53:38,619 INFO [RS:0;jenkins-hbase4:39407] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-25 09:53:38,619 INFO [RS:0;jenkins-hbase4:39407] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-25 09:53:38,619 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1023): About to register with Master. 2023-04-25 09:53:38,623 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33039,1682416415885 with isa=jenkins-hbase4.apache.org/172.31.14.131:39407, startcode=1682416417123 2023-04-25 09:53:38,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-25 09:53:38,636 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 09:53:38,636 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 09:53:38,636 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 09:53:38,636 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-25 09:53:38,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-25 09:53:38,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-25 09:53:38,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,643 DEBUG [RS:0;jenkins-hbase4:39407] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-25 09:53:38,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682416448647 2023-04-25 09:53:38,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-25 09:53:38,653 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-25 09:53:38,654 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-25 09:53:38,662 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-25 09:53:38,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-25 09:53:38,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-25 09:53:38,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-25 09:53:38,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-25 09:53:38,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-25 09:53:38,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,678 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-25 09:53:38,679 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-25 09:53:38,680 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-25 09:53:38,684 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-25 09:53:38,685 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-25 09:53:38,690 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682416418687,5,FailOnTimeoutGroup] 2023-04-25 09:53:38,692 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682416418690,5,FailOnTimeoutGroup] 2023-04-25 09:53:38,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-25 09:53:38,694 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,695 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,711 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-25 09:53:38,713 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-25 09:53:38,713 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c 2023-04-25 09:53:38,722 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56453, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-25 09:53:38,730 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:38,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:38,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-25 09:53:38,756 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/info 2023-04-25 09:53:38,757 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-25 09:53:38,758 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:38,759 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-25 09:53:38,761 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c 2023-04-25 09:53:38,761 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35439 2023-04-25 09:53:38,761 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-25 09:53:38,763 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/rep_barrier 2023-04-25 09:53:38,763 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-25 09:53:38,764 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:38,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-25 09:53:38,767 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 09:53:38,770 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/table 2023-04-25 09:53:38,770 DEBUG [RS:0;jenkins-hbase4:39407] zookeeper.ZKUtil(162): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:38,771 WARN [RS:0;jenkins-hbase4:39407] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-25 09:53:38,772 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-25 09:53:38,772 INFO [RS:0;jenkins-hbase4:39407] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 09:53:38,773 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/WALs/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:38,773 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39407,1682416417123] 2023-04-25 09:53:38,774 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:38,777 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740 2023-04-25 09:53:38,779 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740 2023-04-25 09:53:38,785 DEBUG [RS:0;jenkins-hbase4:39407] zookeeper.ZKUtil(162): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:38,787 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:53:38,790 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-25 09:53:38,795 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:53:38,797 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64679457, jitterRate=-0.03620098531246185}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:53:38,797 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-25 09:53:38,797 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-25 09:53:38,797 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-25 09:53:38,797 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-25 09:53:38,797 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-25 09:53:38,797 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-25 09:53:38,798 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-25 09:53:38,799 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-25 09:53:38,799 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-25 09:53:38,804 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-25 09:53:38,804 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-25 09:53:38,810 INFO [RS:0;jenkins-hbase4:39407] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-25 09:53:38,814 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-25 09:53:38,829 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-25 09:53:38,833 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-25 09:53:38,838 INFO [RS:0;jenkins-hbase4:39407] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-25 09:53:38,842 INFO [RS:0;jenkins-hbase4:39407] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-25 09:53:38,842 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,843 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-25 09:53:38,852 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,852 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,853 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,853 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,853 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,853 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,854 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-25 09:53:38,854 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,854 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,854 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,854 DEBUG [RS:0;jenkins-hbase4:39407] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-25 09:53:38,864 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,864 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,865 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,888 INFO [RS:0;jenkins-hbase4:39407] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-25 09:53:38,891 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39407,1682416417123-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:38,911 INFO [RS:0;jenkins-hbase4:39407] regionserver.Replication(203): jenkins-hbase4.apache.org,39407,1682416417123 started 2023-04-25 09:53:38,912 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39407,1682416417123, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39407, sessionid=0x101babd1af30001 2023-04-25 09:53:38,912 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-25 09:53:38,912 DEBUG [RS:0;jenkins-hbase4:39407] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:38,912 DEBUG [RS:0;jenkins-hbase4:39407] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39407,1682416417123' 2023-04-25 09:53:38,912 DEBUG [RS:0;jenkins-hbase4:39407] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:38,913 DEBUG [RS:0;jenkins-hbase4:39407] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:38,914 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-25 09:53:38,914 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-25 09:53:38,914 DEBUG [RS:0;jenkins-hbase4:39407] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:38,914 DEBUG [RS:0;jenkins-hbase4:39407] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39407,1682416417123' 2023-04-25 09:53:38,914 DEBUG [RS:0;jenkins-hbase4:39407] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-25 09:53:38,915 DEBUG [RS:0;jenkins-hbase4:39407] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-25 09:53:38,915 DEBUG [RS:0;jenkins-hbase4:39407] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-25 09:53:38,915 INFO [RS:0;jenkins-hbase4:39407] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-25 09:53:38,915 INFO [RS:0;jenkins-hbase4:39407] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-25 09:53:38,985 DEBUG [jenkins-hbase4:33039] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-25 09:53:38,988 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39407,1682416417123, state=OPENING 2023-04-25 09:53:38,995 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-25 09:53:38,997 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:38,998 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-25 09:53:39,001 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:53:39,026 INFO [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39407%2C1682416417123, suffix=, logDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/WALs/jenkins-hbase4.apache.org,39407,1682416417123, archiveDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/oldWALs, maxLogs=32 2023-04-25 09:53:39,044 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36589,DS-73b55fbd-4993-4118-898a-5da5bdc68e53,DISK] 2023-04-25 09:53:39,048 INFO [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/WALs/jenkins-hbase4.apache.org,39407,1682416417123/jenkins-hbase4.apache.org%2C39407%2C1682416417123.1682416419029 2023-04-25 09:53:39,049 DEBUG [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36589,DS-73b55fbd-4993-4118-898a-5da5bdc68e53,DISK]] 2023-04-25 09:53:39,186 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:39,189 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-25 09:53:39,192 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-25 09:53:39,203 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-25 09:53:39,204 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-25 09:53:39,207 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39407%2C1682416417123.meta, suffix=.meta, logDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/WALs/jenkins-hbase4.apache.org,39407,1682416417123, archiveDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/oldWALs, maxLogs=32 2023-04-25 09:53:39,223 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36589,DS-73b55fbd-4993-4118-898a-5da5bdc68e53,DISK] 2023-04-25 09:53:39,227 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/WALs/jenkins-hbase4.apache.org,39407,1682416417123/jenkins-hbase4.apache.org%2C39407%2C1682416417123.meta.1682416419208.meta 2023-04-25 09:53:39,227 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36589,DS-73b55fbd-4993-4118-898a-5da5bdc68e53,DISK]] 2023-04-25 09:53:39,228 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:53:39,229 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-25 09:53:39,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-25 09:53:39,250 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-25 09:53:39,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-25 09:53:39,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:39,256 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-25 09:53:39,256 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-25 09:53:39,259 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-25 09:53:39,260 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/info 2023-04-25 09:53:39,260 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/info 2023-04-25 09:53:39,261 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-25 09:53:39,262 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:39,262 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-25 09:53:39,263 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/rep_barrier 2023-04-25 09:53:39,263 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/rep_barrier 2023-04-25 09:53:39,264 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-25 09:53:39,265 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:39,265 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-25 09:53:39,266 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/table 2023-04-25 09:53:39,267 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/table 2023-04-25 09:53:39,267 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-25 09:53:39,268 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:39,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740 2023-04-25 09:53:39,273 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740 2023-04-25 09:53:39,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:53:39,280 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-25 09:53:39,282 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64456369, jitterRate=-0.03952525556087494}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:53:39,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-25 09:53:39,292 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682416419179 2023-04-25 09:53:39,313 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-25 09:53:39,314 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-25 09:53:39,315 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39407,1682416417123, state=OPEN 2023-04-25 09:53:39,317 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-25 09:53:39,317 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-25 09:53:39,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-25 09:53:39,322 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39407,1682416417123 in 316 msec 2023-04-25 09:53:39,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-25 09:53:39,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 509 msec 2023-04-25 09:53:39,333 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 793 msec 2023-04-25 09:53:39,333 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682416419333, completionTime=-1 2023-04-25 09:53:39,333 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-25 09:53:39,334 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-25 09:53:39,404 DEBUG [hconnection-0x5e92c536-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:39,408 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36744, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:39,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-25 09:53:39,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682416479429 2023-04-25 09:53:39,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682416539429 2023-04-25 09:53:39,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 95 msec 2023-04-25 09:53:39,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33039,1682416415885-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:39,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33039,1682416415885-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:39,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33039,1682416415885-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:39,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:33039, period=300000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:39,458 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-25 09:53:39,465 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-25 09:53:39,474 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-25 09:53:39,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-25 09:53:39,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-25 09:53:39,490 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:53:39,494 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:53:39,520 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:39,523 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975 empty. 2023-04-25 09:53:39,523 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:39,524 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-25 09:53:39,947 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-25 09:53:39,949 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4f0d7b338452e6b7e1ce07e6771dd975, NAME => 'hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:53:39,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:39,964 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4f0d7b338452e6b7e1ce07e6771dd975, disabling compactions & flushes 2023-04-25 09:53:39,964 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:39,964 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:39,964 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. after waiting 0 ms 2023-04-25 09:53:39,964 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:39,964 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:39,964 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4f0d7b338452e6b7e1ce07e6771dd975: 2023-04-25 09:53:39,968 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:53:39,982 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682416419971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416419971"}]},"ts":"1682416419971"} 2023-04-25 09:53:40,007 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:53:40,009 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:53:40,014 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416420009"}]},"ts":"1682416420009"} 2023-04-25 09:53:40,017 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-25 09:53:40,026 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4f0d7b338452e6b7e1ce07e6771dd975, ASSIGN}] 2023-04-25 09:53:40,029 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4f0d7b338452e6b7e1ce07e6771dd975, ASSIGN 2023-04-25 09:53:40,031 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4f0d7b338452e6b7e1ce07e6771dd975, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:53:40,182 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4f0d7b338452e6b7e1ce07e6771dd975, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:40,182 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682416420181"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416420181"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416420181"}]},"ts":"1682416420181"} 2023-04-25 09:53:40,188 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4f0d7b338452e6b7e1ce07e6771dd975, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:53:40,348 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:40,349 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4f0d7b338452e6b7e1ce07e6771dd975, NAME => 'hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:53:40,350 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,350 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:40,350 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,350 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,353 INFO [StoreOpener-4f0d7b338452e6b7e1ce07e6771dd975-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,355 DEBUG [StoreOpener-4f0d7b338452e6b7e1ce07e6771dd975-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/info 2023-04-25 09:53:40,356 DEBUG [StoreOpener-4f0d7b338452e6b7e1ce07e6771dd975-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/info 2023-04-25 09:53:40,356 INFO [StoreOpener-4f0d7b338452e6b7e1ce07e6771dd975-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f0d7b338452e6b7e1ce07e6771dd975 columnFamilyName info 2023-04-25 09:53:40,357 INFO [StoreOpener-4f0d7b338452e6b7e1ce07e6771dd975-1] regionserver.HStore(310): Store=4f0d7b338452e6b7e1ce07e6771dd975/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:40,359 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,360 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,365 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:53:40,368 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:53:40,369 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4f0d7b338452e6b7e1ce07e6771dd975; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60405896, jitterRate=-0.09988200664520264}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-25 09:53:40,369 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4f0d7b338452e6b7e1ce07e6771dd975: 2023-04-25 09:53:40,371 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975., pid=6, masterSystemTime=1682416420342 2023-04-25 09:53:40,376 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:40,376 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:53:40,377 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4f0d7b338452e6b7e1ce07e6771dd975, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:40,378 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682416420377"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416420377"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416420377"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416420377"}]},"ts":"1682416420377"} 2023-04-25 09:53:40,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-25 09:53:40,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4f0d7b338452e6b7e1ce07e6771dd975, server=jenkins-hbase4.apache.org,39407,1682416417123 in 193 msec 2023-04-25 09:53:40,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-25 09:53:40,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4f0d7b338452e6b7e1ce07e6771dd975, ASSIGN in 358 msec 2023-04-25 09:53:40,389 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:53:40,390 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416420390"}]},"ts":"1682416420390"} 2023-04-25 09:53:40,392 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-25 09:53:40,396 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:53:40,398 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 919 msec 2023-04-25 09:53:40,490 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-25 09:53:40,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-25 09:53:40,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:53:40,533 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-25 09:53:40,550 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-25 09:53:40,556 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-25 09:53:40,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-25 09:53:40,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-25 09:53:40,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-04-25 09:53:40,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-25 09:53:40,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-25 09:53:40,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.402sec 2023-04-25 09:53:40,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-25 09:53:40,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-25 09:53:40,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-25 09:53:40,615 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33039,1682416415885-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-25 09:53:40,616 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33039,1682416415885-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-25 09:53:40,629 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-25 09:53:40,672 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x373f9c99 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:40,679 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b116c72, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:40,692 DEBUG [hconnection-0x25e7ed72-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:40,703 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:40,714 INFO [Listener at localhost/33705] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:53:40,734 INFO [Listener at localhost/33705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=204, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=170, AvailableMemoryMB=4912 2023-04-25 09:53:40,751 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:53:40,754 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:53:40,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:53:40,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:53:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 09:53:40,770 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:53:40,773 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:53:40,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-25 09:53:40,776 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:40,777 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b empty. 2023-04-25 09:53:40,779 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:40,779 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:53:40,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 09:53:40,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 09:53:41,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 09:53:41,195 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:53:41,198 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6d622ed976f9d94b52b9c4f656801b6b, NAME => 'TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:53:41,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:41,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6d622ed976f9d94b52b9c4f656801b6b, disabling compactions & flushes 2023-04-25 09:53:41,210 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. after waiting 0 ms 2023-04-25 09:53:41,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,210 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,210 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:41,215 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:53:41,217 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416421216"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416421216"}]},"ts":"1682416421216"} 2023-04-25 09:53:41,220 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:53:41,221 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:53:41,222 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416421221"}]},"ts":"1682416421221"} 2023-04-25 09:53:41,224 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 09:53:41,232 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, ASSIGN}] 2023-04-25 09:53:41,234 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, ASSIGN 2023-04-25 09:53:41,236 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:53:41,388 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6d622ed976f9d94b52b9c4f656801b6b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:41,388 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416421387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416421387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416421387"}]},"ts":"1682416421387"} 2023-04-25 09:53:41,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:53:41,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 09:53:41,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6d622ed976f9d94b52b9c4f656801b6b, NAME => 'TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:53:41,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:53:41,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,555 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,558 DEBUG [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A 2023-04-25 09:53:41,558 DEBUG [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A 2023-04-25 09:53:41,559 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:53:41,559 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d622ed976f9d94b52b9c4f656801b6b columnFamilyName A 2023-04-25 09:53:41,560 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.HStore(310): Store=6d622ed976f9d94b52b9c4f656801b6b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:41,560 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,563 DEBUG [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B 2023-04-25 09:53:41,563 DEBUG [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B 2023-04-25 09:53:41,563 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:53:41,563 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d622ed976f9d94b52b9c4f656801b6b columnFamilyName B 2023-04-25 09:53:41,564 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.HStore(310): Store=6d622ed976f9d94b52b9c4f656801b6b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:41,564 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,567 DEBUG [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C 2023-04-25 09:53:41,567 DEBUG [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C 2023-04-25 09:53:41,567 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:53:41,567 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d622ed976f9d94b52b9c4f656801b6b columnFamilyName C 2023-04-25 09:53:41,568 INFO [StoreOpener-6d622ed976f9d94b52b9c4f656801b6b-1] regionserver.HStore(310): Store=6d622ed976f9d94b52b9c4f656801b6b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:53:41,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:53:41,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:41,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:53:41,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6d622ed976f9d94b52b9c4f656801b6b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67043269, jitterRate=-9.77441668510437E-4}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:53:41,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:41,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., pid=11, masterSystemTime=1682416421544 2023-04-25 09:53:41,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:41,593 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6d622ed976f9d94b52b9c4f656801b6b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:41,593 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416421593"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416421593"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416421593"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416421593"}]},"ts":"1682416421593"} 2023-04-25 09:53:41,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-25 09:53:41,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 in 205 msec 2023-04-25 09:53:41,606 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-25 09:53:41,606 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, ASSIGN in 370 msec 2023-04-25 09:53:41,607 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:53:41,608 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416421607"}]},"ts":"1682416421607"} 2023-04-25 09:53:41,610 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 09:53:41,613 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:53:41,616 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 846 msec 2023-04-25 09:53:41,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-25 09:53:41,897 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-25 09:53:41,901 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,906 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,909 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:41,912 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:41,915 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:53:41,918 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:53:41,926 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,930 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,930 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,934 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,935 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,940 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,940 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,944 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,944 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,948 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,949 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,953 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,953 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,981 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,982 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,987 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:41,987 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:53:41,993 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:53:42,005 DEBUG [hconnection-0xfa92e03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,005 DEBUG [hconnection-0x2454dfc8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,013 DEBUG [hconnection-0x6ff4c9b4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,021 DEBUG [hconnection-0x605ba4da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,025 DEBUG [hconnection-0x7d721dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,025 DEBUG [hconnection-0x6260af1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,029 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,029 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,032 DEBUG [hconnection-0x3e1787eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,037 DEBUG [hconnection-0x5ad720d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,040 DEBUG [hconnection-0x7a95e8d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:53:42,047 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,047 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,048 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:42,063 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36834, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,064 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,065 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,079 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:53:42,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:42,081 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:42,082 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:42,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:42,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:42,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:42,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,092 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:42,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:42,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:42,102 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:42,105 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:42,106 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-25 09:53:42,132 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:42,133 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:42,133 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:42,133 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:42,133 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:42,134 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:42,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:42,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:42,250 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/fc1c20d5f6a044a9b9a3af9457388329 2023-04-25 09:53:42,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416422212, max timestamp=1682416422267Num uniques 859; 2023-04-25 09:53:42,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416482299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416422212, max timestamp=1682416422267Num uniques 859; 2023-04-25 09:53:42,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:53:42,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416422212, max timestamp=1682416422267Num uniques 859; 2023-04-25 09:53:42,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:53:42,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:53:42,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682416422360, max timestamp=1682416422412Num uniques 1009; 2023-04-25 09:53:42,437 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=14558, suffix heap size=132904,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=318 2023-04-25 09:53:42,444 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/2d5ff2eaaa424de0aa8938e2b6a59c17 2023-04-25 09:53:42,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416422374, max timestamp=1682416422412Num uniques 1009; 2023-04-25 09:53:42,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-25 09:53:42,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416422374, max timestamp=1682416422412Num uniques 1009; 2023-04-25 09:53:42,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-25 09:53:42,537 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/e7f80737e8ae4d40ac9bed23c51273da 2023-04-25 09:53:42,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682416422505, max timestamp=1682416422543Num uniques 1027; 2023-04-25 09:53:42,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-25 09:53:42,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682416422505, max timestamp=1682416422543Num uniques 1177; 2023-04-25 09:53:42,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=17262, suffix heap size=150376,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=377 2023-04-25 09:53:42,562 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/fc1c20d5f6a044a9b9a3af9457388329 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fc1c20d5f6a044a9b9a3af9457388329 2023-04-25 09:53:42,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1682416422505, max timestamp=1682416422543Num uniques 1027; 2023-04-25 09:53:42,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=17262, suffix heap size=134776,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=377 2023-04-25 09:53:42,609 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fc1c20d5f6a044a9b9a3af9457388329, entries=150, sequenceid=8, filesize=11.6 K 2023-04-25 09:53:42,614 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/2d5ff2eaaa424de0aa8938e2b6a59c17 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2d5ff2eaaa424de0aa8938e2b6a59c17 2023-04-25 09:53:42,642 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2d5ff2eaaa424de0aa8938e2b6a59c17, entries=150, sequenceid=8, filesize=11.6 K 2023-04-25 09:53:42,645 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/e7f80737e8ae4d40ac9bed23c51273da as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e7f80737e8ae4d40ac9bed23c51273da 2023-04-25 09:53:42,657 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e7f80737e8ae4d40ac9bed23c51273da, entries=150, sequenceid=8, filesize=11.6 K 2023-04-25 09:53:42,659 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=107.34 KB/109920 for 6d622ed976f9d94b52b9c4f656801b6b in 553ms, sequenceid=8, compaction requested=false 2023-04-25 09:53:42,661 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 09:53:42,663 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:42,663 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:42,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:42,669 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:42,669 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:42,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:42,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@631726a2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:42,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:42,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:42,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=114.05 KB heapSize=304.78 KB 2023-04-25 09:53:42,677 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:42,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:42,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:42,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:42,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:42,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:42,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:42,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:42,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:42,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:42,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:42,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:42,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:42,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:42,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:42,696 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:42,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:42,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:42,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:42,696 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:42,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:42,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:42,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:42,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:42,710 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,714 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:42,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:42,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:42,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:42,739 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:42,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:42,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:42,743 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:42,743 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:42,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416482714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416482715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416482717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416482718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416482722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/4011e030d0744b169570be7927b3f212 2023-04-25 09:53:42,843 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:42,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:42,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/dc9a99cbc973449c97fd6e4d899d9539 2023-04-25 09:53:42,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:42,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416482859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416482860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416482861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416482862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:42,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416482863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:42,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:42,887 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:42,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:42,893 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:42,893 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:42,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,896 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:42,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:42,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,898 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,898 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:42,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:42,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:42,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:42,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:42,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:42,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:42,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:42,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:42,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:42,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,909 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:42,909 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:42,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:42,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:42,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298d285a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:42,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:42,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,923 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:42,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:42,930 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:42,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:42,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:42,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:42,938 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:42,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:42,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:42,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:42,938 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:42,943 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:42,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:42,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:42,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:42,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:42,967 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:42,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:42,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:42,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:42,970 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:42,970 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:42,970 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,970 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:42,970 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:42,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:42,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:42,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:42,971 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:42,971 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:43,071 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:43,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:43,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416483070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416483070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416483072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416483072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416483075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:43,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:43,085 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:43,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,090 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:43,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:43,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,094 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:43,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:43,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:43,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:43,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:43,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:43,103 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,103 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:43,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:43,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:43,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:43,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:43,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:43,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:43,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c09dacc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:43,116 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:43,116 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:43,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:43,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:43,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:43,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:43,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:43,126 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:43,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:43,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:43,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:43,127 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:43,131 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,131 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,131 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,134 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,138 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:43,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:43,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:43,160 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:43,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,260 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:43,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:43,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:43,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:43,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:43,271 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:43,271 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,273 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:43,273 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:43,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:43,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:43,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:43,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:43,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:43,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:43,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:43,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:43,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:43,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:43,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:43,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:43,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:43,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/232da319eb2f49dc9a6946351f3d607b 2023-04-25 09:53:43,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:43,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50156cf3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:43,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:43,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,321 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,321 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:43,333 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,333 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,333 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,334 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:43,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/4011e030d0744b169570be7927b3f212 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4011e030d0744b169570be7927b3f212 2023-04-25 09:53:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:43,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:43,342 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:43,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:43,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:43,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:43,342 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:43,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4011e030d0744b169570be7927b3f212, entries=400, sequenceid=72, filesize=23.0 K 2023-04-25 09:53:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/dc9a99cbc973449c97fd6e4d899d9539 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/dc9a99cbc973449c97fd6e4d899d9539 2023-04-25 09:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,366 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,367 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:43,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,370 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:43,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:43,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/dc9a99cbc973449c97fd6e4d899d9539, entries=150, sequenceid=72, filesize=11.6 K 2023-04-25 09:53:43,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/232da319eb2f49dc9a6946351f3d607b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/232da319eb2f49dc9a6946351f3d607b 2023-04-25 09:53:43,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416483384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/232da319eb2f49dc9a6946351f3d607b, entries=150, sequenceid=72, filesize=11.6 K 2023-04-25 09:53:43,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416483384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~304.73 KB/312048, currentSize=73.80 KB/75570 for 6d622ed976f9d94b52b9c4f656801b6b in 738ms, sequenceid=72, compaction requested=false 2023-04-25 09:53:43,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:43,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:43,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:53:43,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:43,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:43,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:43,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:43,449 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:43,449 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:43,470 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:43,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:43,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:43,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:43,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:43,488 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:43,488 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,494 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:43,494 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:43,495 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,495 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,497 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:43,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:43,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:43,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:43,508 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:43,508 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,509 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:43,511 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:43,511 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:43,511 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,511 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:43,511 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:43,512 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,514 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:43,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:43,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416483498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:43,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:43,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416483500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416483515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,532 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,534 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:43,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aed6265[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:43,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:43,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,542 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:43,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:43,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:43,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:43,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/0f12414a98aa43ae8176fc4f49af6166 2023-04-25 09:53:43,545 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:43,545 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,545 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:43,545 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:43,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:43,559 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:43,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:43,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:43,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:43,559 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:43,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:43,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:43,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:43,584 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:43,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1406e0bb5af6441a88aa03a10305b840 2023-04-25 09:53:43,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416483628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416483629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416483630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:43,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:43,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:43,728 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:43,728 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,729 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:43,729 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:43,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,732 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:43,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:43,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:43,737 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,737 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:43,737 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:43,737 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:43,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/0276ad2b88c1432d8ec0dc54c25cc3c3 2023-04-25 09:53:43,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,750 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a62549a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:43,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:43,755 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,757 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:43,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:43,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:43,757 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,757 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:43,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/0f12414a98aa43ae8176fc4f49af6166 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0f12414a98aa43ae8176fc4f49af6166 2023-04-25 09:53:43,761 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:43,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:43,766 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:43,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:43,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:43,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:43,766 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:43,769 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,769 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,769 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,774 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0f12414a98aa43ae8176fc4f49af6166, entries=250, sequenceid=92, filesize=16.2 K 2023-04-25 09:53:43,779 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,779 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,779 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:43,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:43,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1406e0bb5af6441a88aa03a10305b840 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1406e0bb5af6441a88aa03a10305b840 2023-04-25 09:53:43,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:43,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:43,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:43,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:43,781 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,781 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,781 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,782 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,782 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,781 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:43,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:43,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:43,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1406e0bb5af6441a88aa03a10305b840, entries=150, sequenceid=92, filesize=11.6 K 2023-04-25 09:53:43,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/0276ad2b88c1432d8ec0dc54c25cc3c3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0276ad2b88c1432d8ec0dc54c25cc3c3 2023-04-25 09:53:43,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0276ad2b88c1432d8ec0dc54c25cc3c3, entries=150, sequenceid=92, filesize=11.6 K 2023-04-25 09:53:43,831 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6d622ed976f9d94b52b9c4f656801b6b in 386ms, sequenceid=92, compaction requested=true 2023-04-25 09:53:43,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:43,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:43,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:43,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:43,859 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:43,859 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:43,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:43,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 09:53:43,863 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:43,865 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:43,866 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,866 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2d5ff2eaaa424de0aa8938e2b6a59c17, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/dc9a99cbc973449c97fd6e4d899d9539, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1406e0bb5af6441a88aa03a10305b840] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=34.8 K 2023-04-25 09:53:43,868 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2d5ff2eaaa424de0aa8938e2b6a59c17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1682416422066 2023-04-25 09:53:43,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:43,883 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:43,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:43,872 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting dc9a99cbc973449c97fd6e4d899d9539, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682416422605 2023-04-25 09:53:43,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:43,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:43,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:43,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:43,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:43,886 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1406e0bb5af6441a88aa03a10305b840, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682416423418 2023-04-25 09:53:43,889 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 52042 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:43,889 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:43,889 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,890 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fc1c20d5f6a044a9b9a3af9457388329, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4011e030d0744b169570be7927b3f212, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0f12414a98aa43ae8176fc4f49af6166] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=50.8 K 2023-04-25 09:53:43,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:43,906 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting fc1c20d5f6a044a9b9a3af9457388329, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1682416422066 2023-04-25 09:53:43,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:43,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:43,921 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:43,922 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,922 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:43,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:43,926 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4011e030d0744b169570be7927b3f212, keycount=400, bloomtype=ROW, size=23.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682416422600 2023-04-25 09:53:43,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,928 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0f12414a98aa43ae8176fc4f49af6166, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682416422716 2023-04-25 09:53:43,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,929 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:43,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:43,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,930 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#10 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:43,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:43,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:43,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:43,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:43,969 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:43,969 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:43,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:43,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:43,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,997 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416483963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:44,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:44,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@285f38a0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:44,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416483966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416483971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:44,009 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#11 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:44,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416483985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416483971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:44,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:44,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:44,026 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:44,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:44,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:44,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:44,026 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:44,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:44,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:44,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,040 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:44,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,041 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,057 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/015c83af25fd4fd9a64e04fdf56b66db as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/015c83af25fd4fd9a64e04fdf56b66db 2023-04-25 09:53:44,095 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 015c83af25fd4fd9a64e04fdf56b66db(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:44,095 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:44,095 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=13, startTime=1682416423858; duration=0sec 2023-04-25 09:53:44,098 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:44,099 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:44,102 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:44,102 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:44,102 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,103 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e7f80737e8ae4d40ac9bed23c51273da, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/232da319eb2f49dc9a6946351f3d607b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0276ad2b88c1432d8ec0dc54c25cc3c3] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=34.8 K 2023-04-25 09:53:44,104 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e7f80737e8ae4d40ac9bed23c51273da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1682416422066 2023-04-25 09:53:44,105 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 232da319eb2f49dc9a6946351f3d607b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682416422605 2023-04-25 09:53:44,108 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0276ad2b88c1432d8ec0dc54c25cc3c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682416423418 2023-04-25 09:53:44,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,118 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/401df10a3760489aba74156476d7ae90 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/401df10a3760489aba74156476d7ae90 2023-04-25 09:53:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416484113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416484111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416484113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416484117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416484130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,141 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:44,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:44,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:44,184 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#12 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:44,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:44,188 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:44,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,191 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 401df10a3760489aba74156476d7ae90(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:44,191 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:44,212 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=13, startTime=1682416423833; duration=0sec 2023-04-25 09:53:44,212 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:44,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:44,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:44,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,215 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,216 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,216 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:44,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:44,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:44,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:44,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:44,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:44,295 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416484326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416484327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416484327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416484367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,367 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5356abac[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:44,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:44,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,391 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/54ffc61544674c71a32eba54f5ebde47 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/54ffc61544674c71a32eba54f5ebde47 2023-04-25 09:53:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416484371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:44,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,393 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:44,393 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:44,393 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:44,395 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:44,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:44,403 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:44,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:44,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:44,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:44,403 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:44,404 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 54ffc61544674c71a32eba54f5ebde47(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:44,404 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:44,404 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=13, startTime=1682416423858; duration=0sec 2023-04-25 09:53:44,404 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:44,413 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,413 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/648018cbdbfc4684a0f543d458c3d7f7 2023-04-25 09:53:44,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:44,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:44,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,429 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:44,429 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:44,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:44,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1ab32767dc1b4b7eb27d443a98695964 2023-04-25 09:53:44,530 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:44,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:44,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:44,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:44,539 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:44,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:44,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:44,541 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,541 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,545 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:44,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:44,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:44,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,584 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:44,616 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:44,616 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,616 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:44,616 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:44,616 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:44,617 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:44,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:44,623 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,624 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,624 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:44,624 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:44,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c905e9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:44,627 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:44,627 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,630 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:44,631 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:44,631 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:44,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:44,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,648 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416484642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416484642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:44,657 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:44,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:44,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:44,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:44,657 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:44,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416484652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:44,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:44,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:44,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,668 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:44,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,668 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:44,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416484672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,677 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:44,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416484697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/5d0f727d3263451b954450ce993fe8b9 2023-04-25 09:53:44,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/648018cbdbfc4684a0f543d458c3d7f7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/648018cbdbfc4684a0f543d458c3d7f7 2023-04-25 09:53:44,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/648018cbdbfc4684a0f543d458c3d7f7, entries=150, sequenceid=106, filesize=11.6 K 2023-04-25 09:53:44,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1ab32767dc1b4b7eb27d443a98695964 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1ab32767dc1b4b7eb27d443a98695964 2023-04-25 09:53:44,768 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:44,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:44,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:44,780 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:44,780 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:44,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:44,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:44,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:44,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:44,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:44,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:44,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:44,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:44,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:44,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:44,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:44,804 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1ab32767dc1b4b7eb27d443a98695964, entries=150, sequenceid=106, filesize=11.6 K 2023-04-25 09:53:44,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:44,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/5d0f727d3263451b954450ce993fe8b9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5d0f727d3263451b954450ce993fe8b9 2023-04-25 09:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:44,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5751ae[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:44,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:44,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,814 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:44,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:44,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:44,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:44,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,822 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:44,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5d0f727d3263451b954450ce993fe8b9, entries=150, sequenceid=106, filesize=11.6 K 2023-04-25 09:53:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6d622ed976f9d94b52b9c4f656801b6b in 966ms, sequenceid=106, compaction requested=false 2023-04-25 09:53:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:44,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:44,832 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:44,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:44,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:44,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:44,832 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:44,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,836 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:44,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:44,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:44,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:44,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:44,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:44,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:44,852 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:44,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:44,952 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:44,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:44,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:44,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:44,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:44,964 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:44,964 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:44,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:44,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,969 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:44,969 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:44,970 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,970 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:44,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:44,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:44,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:44,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:44,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:44,987 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:44,988 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:45,012 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 09:53:45,105 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-25 09:53:45,106 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-25 09:53:45,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:45,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:45,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416485190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416485198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416485201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416485214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416485220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416485306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416485306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416485307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,416 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/cbe2e5436aa0493cbcdfef9cd092f8ca 2023-04-25 09:53:45,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416485513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416485514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416485514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416485822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416485822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:45,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416485822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:45,868 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/23ea49e422b44ba3857397c1e2f8c33b 2023-04-25 09:53:46,053 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 09:53:46,054 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-25 09:53:46,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:46,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416486223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:46,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416486231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416486327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416486336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416486337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,366 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/2a4f5e7f0fb94b3092ba43a4ed733e1a 2023-04-25 09:53:46,386 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/cbe2e5436aa0493cbcdfef9cd092f8ca as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cbe2e5436aa0493cbcdfef9cd092f8ca 2023-04-25 09:53:46,402 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cbe2e5436aa0493cbcdfef9cd092f8ca, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 09:53:46,407 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/23ea49e422b44ba3857397c1e2f8c33b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/23ea49e422b44ba3857397c1e2f8c33b 2023-04-25 09:53:46,423 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/23ea49e422b44ba3857397c1e2f8c33b, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 09:53:46,429 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/2a4f5e7f0fb94b3092ba43a4ed733e1a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2a4f5e7f0fb94b3092ba43a4ed733e1a 2023-04-25 09:53:46,438 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2a4f5e7f0fb94b3092ba43a4ed733e1a, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 09:53:46,440 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6d622ed976f9d94b52b9c4f656801b6b in 1454ms, sequenceid=127, compaction requested=true 2023-04-25 09:53:46,440 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:46,440 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:46,440 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:46,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:46,442 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:46,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:46,447 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:46,447 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,447 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:46,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b9cee7c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:46,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:46,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,453 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:46,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:46,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:46,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:46,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:46,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:46,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:46,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:46,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:46,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:46,461 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:46,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:46,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:46,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:46,462 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:46,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:46,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:46,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:46,468 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:46,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:46,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:46,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:46,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:46,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:46,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:46,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:46,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:46,476 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:46,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,477 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:46,477 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,477 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:46,478 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:46,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:46,577 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:46,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:46,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:46,608 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:46,608 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:46,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:46,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:46,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:46,614 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:46,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:46,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:46,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:46,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:46,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:46,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:46,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:46,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:46,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:46,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:46,644 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:53:46,652 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:46,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:46,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:46,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:46,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:46,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,139 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/852d58246241480692d5d892afcf9544 2023-04-25 09:53:47,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:47,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:47,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416487376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416487376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416487378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416487480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416487483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416487483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,580 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/826271dca455474aada2acae3ecc708b 2023-04-25 09:53:47,607 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/7adc6288e57f4ca89caa92f7a9869150 2023-04-25 09:53:47,619 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/852d58246241480692d5d892afcf9544 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/852d58246241480692d5d892afcf9544 2023-04-25 09:53:47,627 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/852d58246241480692d5d892afcf9544, entries=150, sequenceid=141, filesize=11.8 K 2023-04-25 09:53:47,630 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/826271dca455474aada2acae3ecc708b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/826271dca455474aada2acae3ecc708b 2023-04-25 09:53:47,637 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/826271dca455474aada2acae3ecc708b, entries=150, sequenceid=141, filesize=11.8 K 2023-04-25 09:53:47,640 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/7adc6288e57f4ca89caa92f7a9869150 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7adc6288e57f4ca89caa92f7a9869150 2023-04-25 09:53:47,659 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7adc6288e57f4ca89caa92f7a9869150, entries=150, sequenceid=141, filesize=11.8 K 2023-04-25 09:53:47,662 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6d622ed976f9d94b52b9c4f656801b6b in 1018ms, sequenceid=141, compaction requested=true 2023-04-25 09:53:47,662 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:47,662 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:47,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:47,663 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:47,663 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:47,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:47,669 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:47,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df13968[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:47,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:47,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:47,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:47,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:47,682 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:47,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:47,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:47,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:47,682 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:47,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:47,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:47,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,690 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:47,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:47,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:47,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:47,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:47,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:47,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:47,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:47,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:47,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:47,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:47,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:47,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/d7224f3b66ed4ab59ef8340ee1d2fc47 2023-04-25 09:53:47,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416487715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416487716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416487718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/5c50fc50352b4481a30dab60271e9c09 2023-04-25 09:53:47,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/369c647950964c7a9730f3e8efdae3b2 2023-04-25 09:53:47,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/d7224f3b66ed4ab59ef8340ee1d2fc47 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/d7224f3b66ed4ab59ef8340ee1d2fc47 2023-04-25 09:53:47,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/d7224f3b66ed4ab59ef8340ee1d2fc47, entries=200, sequenceid=161, filesize=14.1 K 2023-04-25 09:53:47,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/5c50fc50352b4481a30dab60271e9c09 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/5c50fc50352b4481a30dab60271e9c09 2023-04-25 09:53:47,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:47,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:47,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:47,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/5c50fc50352b4481a30dab60271e9c09, entries=150, sequenceid=161, filesize=11.8 K 2023-04-25 09:53:47,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/369c647950964c7a9730f3e8efdae3b2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/369c647950964c7a9730f3e8efdae3b2 2023-04-25 09:53:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:47,817 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:47,817 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:47,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:47,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:47,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:47,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:47,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:47,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:47,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:47,824 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:47,824 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:47,824 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:47,824 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:47,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416487819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:47,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416487820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/369c647950964c7a9730f3e8efdae3b2, entries=150, sequenceid=161, filesize=11.8 K 2023-04-25 09:53:47,827 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6d622ed976f9d94b52b9c4f656801b6b in 139ms, sequenceid=161, compaction requested=true 2023-04-25 09:53:47,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:47,827 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:47,827 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:53:47,827 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:47,827 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:53:47,827 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,833 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,833 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62223 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:53:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:47,833 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:47,833 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:47,833 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/401df10a3760489aba74156476d7ae90, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/648018cbdbfc4684a0f543d458c3d7f7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cbe2e5436aa0493cbcdfef9cd092f8ca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/852d58246241480692d5d892afcf9544, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/d7224f3b66ed4ab59ef8340ee1d2fc47] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=60.8 K 2023-04-25 09:53:47,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:47,835 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:53:47,835 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:47,835 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:47,835 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/015c83af25fd4fd9a64e04fdf56b66db, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1ab32767dc1b4b7eb27d443a98695964, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/23ea49e422b44ba3857397c1e2f8c33b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/826271dca455474aada2acae3ecc708b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/5c50fc50352b4481a30dab60271e9c09] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=58.4 K 2023-04-25 09:53:47,835 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 401df10a3760489aba74156476d7ae90, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682416423418 2023-04-25 09:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:47,840 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 015c83af25fd4fd9a64e04fdf56b66db, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682416423418 2023-04-25 09:53:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:47,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 09:53:47,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:47,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,841 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 648018cbdbfc4684a0f543d458c3d7f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682416423510 2023-04-25 09:53:47,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:47,841 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1ab32767dc1b4b7eb27d443a98695964, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682416423510 2023-04-25 09:53:47,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:47,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:47,842 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting cbe2e5436aa0493cbcdfef9cd092f8ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416423944 2023-04-25 09:53:47,842 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 23ea49e422b44ba3857397c1e2f8c33b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416423944 2023-04-25 09:53:47,843 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 852d58246241480692d5d892afcf9544, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682416425176 2023-04-25 09:53:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,844 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting d7224f3b66ed4ab59ef8340ee1d2fc47, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682416427367 2023-04-25 09:53:47,844 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 826271dca455474aada2acae3ecc708b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682416425176 2023-04-25 09:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:47,846 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5c50fc50352b4481a30dab60271e9c09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682416427367 2023-04-25 09:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ed8972[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:47,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:47,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,848 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:47,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:47,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:47,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:47,857 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:47,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:47,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:47,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:47,857 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:47,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:47,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:47,863 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:47,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:47,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:47,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:47,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:47,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:47,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,872 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,872 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:47,872 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:47,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:47,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:47,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:47,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:47,896 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#25 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:47,903 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#26 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:47,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/60c9e7a28d384c55be35531ec663fc08 2023-04-25 09:53:47,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/0dcddf69e47440bc9500e89a0b79de35 2023-04-25 09:53:47,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:47,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:47,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:47,989 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:47,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:47,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:47,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:47,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,996 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:47,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:47,997 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:47,997 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:47,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:48,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:48,002 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,002 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:48,002 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:48,002 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:48,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,004 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,004 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77677600[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:48,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:48,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,012 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:48,012 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:48,012 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:48,012 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,012 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:48,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416488013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:48,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:48,026 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:48,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:48,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:48,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:48,026 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:48,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416488028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,030 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,032 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:48,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:48,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,036 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,037 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:48,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:48,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416488033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416488126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,138 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:48,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:48,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:48,144 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:48,144 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:48,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:48,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,147 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:48,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:48,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:48,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:48,150 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,150 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:48,150 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:48,150 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd4999e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:48,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:48,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:48,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:48,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:48,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:48,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:48,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:48,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:48,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:48,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:48,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:48,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:48,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:48,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:48,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416488238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,240 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4269 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:48,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416488264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,286 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4296 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:48,288 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:48,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:48,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:48,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:48,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:48,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:48,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:48,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:48,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,303 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,303 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:48,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:48,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:48,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:48,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,306 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:48,307 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,307 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:48,307 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:48,307 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,307 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:48,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:48,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbadff5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:48,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:48,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:48,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:48,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:48,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:48,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:48,327 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:48,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:48,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:48,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:48,327 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:48,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,332 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416488331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416488332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:48,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:48,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:48,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:48,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:48,338 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,338 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,338 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416488341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,349 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/d5d97552dc5f4629983806943640effe as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/d5d97552dc5f4629983806943640effe 2023-04-25 09:53:48,349 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/61b72f31b68c416abfdad9320374d1ef as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/61b72f31b68c416abfdad9320374d1ef 2023-04-25 09:53:48,358 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 61b72f31b68c416abfdad9320374d1ef(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:48,358 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:48,359 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=11, startTime=1682416427827; duration=0sec 2023-04-25 09:53:48,359 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:48,359 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:53:48,359 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into d5d97552dc5f4629983806943640effe(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:48,359 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:48,359 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=11, startTime=1682416427827; duration=0sec 2023-04-25 09:53:48,360 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:48,361 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:53:48,362 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:48,362 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,362 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/54ffc61544674c71a32eba54f5ebde47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5d0f727d3263451b954450ce993fe8b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2a4f5e7f0fb94b3092ba43a4ed733e1a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7adc6288e57f4ca89caa92f7a9869150, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/369c647950964c7a9730f3e8efdae3b2] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=58.4 K 2023-04-25 09:53:48,362 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 54ffc61544674c71a32eba54f5ebde47, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1682416423418 2023-04-25 09:53:48,364 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5d0f727d3263451b954450ce993fe8b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1682416423510 2023-04-25 09:53:48,365 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2a4f5e7f0fb94b3092ba43a4ed733e1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416423944 2023-04-25 09:53:48,365 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 7adc6288e57f4ca89caa92f7a9869150, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1682416425176 2023-04-25 09:53:48,366 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 369c647950964c7a9730f3e8efdae3b2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682416427367 2023-04-25 09:53:48,386 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#29 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:48,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/96ef4e23c9d146e49d86626d170407c7 2023-04-25 09:53:48,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/60c9e7a28d384c55be35531ec663fc08 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60c9e7a28d384c55be35531ec663fc08 2023-04-25 09:53:48,421 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60c9e7a28d384c55be35531ec663fc08, entries=200, sequenceid=174, filesize=14.1 K 2023-04-25 09:53:48,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/0dcddf69e47440bc9500e89a0b79de35 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0dcddf69e47440bc9500e89a0b79de35 2023-04-25 09:53:48,424 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/34fc2da8903b441c9bf18d310e131127 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/34fc2da8903b441c9bf18d310e131127 2023-04-25 09:53:48,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:48,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:48,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:48,443 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 34fc2da8903b441c9bf18d310e131127(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:48,443 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:48,443 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=11, startTime=1682416427827; duration=0sec 2023-04-25 09:53:48,443 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:48,447 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:48,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0dcddf69e47440bc9500e89a0b79de35, entries=150, sequenceid=174, filesize=11.8 K 2023-04-25 09:53:48,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:48,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:48,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/96ef4e23c9d146e49d86626d170407c7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/96ef4e23c9d146e49d86626d170407c7 2023-04-25 09:53:48,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,450 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,451 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,451 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:48,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/96ef4e23c9d146e49d86626d170407c7, entries=150, sequenceid=174, filesize=11.8 K 2023-04-25 09:53:48,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:48,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:48,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:48,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:48,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6d622ed976f9d94b52b9c4f656801b6b in 624ms, sequenceid=174, compaction requested=false 2023-04-25 09:53:48,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:48,467 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34576f69[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:48,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:48,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:48,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:48,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:48,485 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:48,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:48,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:48,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:48,485 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:48,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,489 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,491 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,495 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,495 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,495 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,495 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,495 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:48,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,495 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,496 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,497 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,497 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:48,497 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,497 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:48,497 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:48,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:48,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:48,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:48,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:48,603 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:48,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:48,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:48,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,616 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:48,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:48,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:48,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:48,648 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,648 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:48,648 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:53:48,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:48,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:48,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:48,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:48,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:48,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:48,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:48,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:48,706 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/85b82c74a2614c36b931bc2dc4091164 2023-04-25 09:53:48,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416488721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,785 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/063ab73ce6f546f69c667c89bea7aee5 2023-04-25 09:53:48,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36798 deadline: 1682416488826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416488837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416488845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,848 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/b7a7c369f20248f394ca63233aef3bce 2023-04-25 09:53:48,858 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/85b82c74a2614c36b931bc2dc4091164 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/85b82c74a2614c36b931bc2dc4091164 2023-04-25 09:53:48,869 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/85b82c74a2614c36b931bc2dc4091164, entries=200, sequenceid=197, filesize=14.1 K 2023-04-25 09:53:48,871 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/063ab73ce6f546f69c667c89bea7aee5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/063ab73ce6f546f69c667c89bea7aee5 2023-04-25 09:53:48,878 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/063ab73ce6f546f69c667c89bea7aee5, entries=150, sequenceid=197, filesize=11.8 K 2023-04-25 09:53:48,880 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/b7a7c369f20248f394ca63233aef3bce as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/b7a7c369f20248f394ca63233aef3bce 2023-04-25 09:53:48,895 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/b7a7c369f20248f394ca63233aef3bce, entries=150, sequenceid=197, filesize=11.8 K 2023-04-25 09:53:48,897 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6d622ed976f9d94b52b9c4f656801b6b in 249ms, sequenceid=197, compaction requested=true 2023-04-25 09:53:48,897 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:48,897 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,901 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,902 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ccc895[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:48,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:48,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,907 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:48,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:48,907 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,907 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:48,909 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,910 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:48,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:48,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:48,923 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:48,923 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:48,923 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:48,923 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:48,923 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:48,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:48,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:48,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:48,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:48,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:48,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:48,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:48,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:48,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:48,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:48,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:48,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:48,945 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:48,945 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:48,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:49,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:49,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 09:53:49,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:49,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:49,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:49,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:49,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:49,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:49,047 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,080 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,080 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@775bb67b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,097 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:49,106 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:49,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:49,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:49,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:49,106 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:49,108 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,108 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,108 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,110 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:49,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416489110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:49,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:49,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:49,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,117 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:49,117 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:49,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,217 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:49,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416489215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,223 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,235 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,236 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,236 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,236 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,236 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,241 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eb79a12[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,246 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,246 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,246 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,249 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:49,265 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:49,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:49,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:49,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:49,265 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:49,274 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,274 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,274 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,276 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:49,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:49,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:49,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:49,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,391 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,394 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,405 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,405 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,408 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51149a8a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,414 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,416 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:49,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:49,420 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:49,420 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:49,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,423 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:49,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416489421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:49,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:49,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,431 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:49,431 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:49,431 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/0e375922354947dbbc87c1ce2506319f 2023-04-25 09:53:49,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/2113dc558b64475686c139afad4fa6fb 2023-04-25 09:53:49,531 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,539 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,540 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,548 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,551 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f82685d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,556 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:49,566 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:49,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:49,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:49,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:49,567 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:49,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,571 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:49,577 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:49,577 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:49,580 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,580 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,677 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,682 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,684 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,687 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,687 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,687 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,687 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6540f7b8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,698 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,698 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:49,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:49,704 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:49,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:49,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:49,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:49,704 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:49,709 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,709 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,710 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,714 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:49,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:49,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,715 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:49,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:49,716 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,716 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:49,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416489728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,824 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,829 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,833 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,835 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,836 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,836 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,836 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,836 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c58608[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,842 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,842 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,843 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416489848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:49,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:49,851 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:49,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:49,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:49,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:49,851 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:49,853 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,853 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416489854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,861 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:49,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:49,868 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:49,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:49,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:49,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/16bbe43a21984646b3584443c056dbff 2023-04-25 09:53:49,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/0e375922354947dbbc87c1ce2506319f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0e375922354947dbbc87c1ce2506319f 2023-04-25 09:53:49,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0e375922354947dbbc87c1ce2506319f, entries=150, sequenceid=210, filesize=11.8 K 2023-04-25 09:53:49,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/2113dc558b64475686c139afad4fa6fb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2113dc558b64475686c139afad4fa6fb 2023-04-25 09:53:49,968 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:49,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:49,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:49,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:49,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:49,973 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:49,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:49,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:49,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,975 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:49,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:49,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:49,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:49,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:49,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:49,980 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,980 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:49,980 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:49,980 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:49,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:49,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2113dc558b64475686c139afad4fa6fb, entries=150, sequenceid=210, filesize=11.8 K 2023-04-25 09:53:49,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:49,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/16bbe43a21984646b3584443c056dbff as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/16bbe43a21984646b3584443c056dbff 2023-04-25 09:53:49,984 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72708cb0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:49,986 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:49,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,990 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:49,990 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:49,990 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:49,990 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:49,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:49,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:49,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/16bbe43a21984646b3584443c056dbff, entries=150, sequenceid=210, filesize=11.8 K 2023-04-25 09:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:49,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6d622ed976f9d94b52b9c4f656801b6b in 961ms, sequenceid=210, compaction requested=true 2023-04-25 09:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:49,997 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:53:49,997 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:53:49,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:49,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,001 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53198 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:53:50,001 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:50,001 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,001 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/61b72f31b68c416abfdad9320374d1ef, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60c9e7a28d384c55be35531ec663fc08, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/85b82c74a2614c36b931bc2dc4091164, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0e375922354947dbbc87c1ce2506319f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=52.0 K 2023-04-25 09:53:50,002 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:53:50,002 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:50,002 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,003 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/d5d97552dc5f4629983806943640effe, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0dcddf69e47440bc9500e89a0b79de35, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/063ab73ce6f546f69c667c89bea7aee5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2113dc558b64475686c139afad4fa6fb] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=47.3 K 2023-04-25 09:53:50,003 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d5d97552dc5f4629983806943640effe, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682416427367 2023-04-25 09:53:50,003 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 61b72f31b68c416abfdad9320374d1ef, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682416427367 2023-04-25 09:53:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:50,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:50,005 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:50,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:50,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:50,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:50,005 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:50,005 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0dcddf69e47440bc9500e89a0b79de35, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1682416427701 2023-04-25 09:53:50,006 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 063ab73ce6f546f69c667c89bea7aee5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682416427976 2023-04-25 09:53:50,006 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 60c9e7a28d384c55be35531ec663fc08, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1682416427701 2023-04-25 09:53:50,008 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2113dc558b64475686c139afad4fa6fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682416428709 2023-04-25 09:53:50,008 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 85b82c74a2614c36b931bc2dc4091164, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682416427976 2023-04-25 09:53:50,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,012 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0e375922354947dbbc87c1ce2506319f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682416428709 2023-04-25 09:53:50,013 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,023 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:50,023 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:50,023 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:50,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:50,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:50,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:50,024 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:50,024 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:50,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:50,031 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:50,031 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#36 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:50,086 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/fa1109b351f040eb95ec26e46e9d6d7a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fa1109b351f040eb95ec26e46e9d6d7a 2023-04-25 09:53:50,100 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into fa1109b351f040eb95ec26e46e9d6d7a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:50,100 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:50,100 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=12, startTime=1682416429997; duration=0sec 2023-04-25 09:53:50,101 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:50,101 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:53:50,106 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:53:50,106 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:50,106 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,107 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/34fc2da8903b441c9bf18d310e131127, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/96ef4e23c9d146e49d86626d170407c7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/b7a7c369f20248f394ca63233aef3bce, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/16bbe43a21984646b3584443c056dbff] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=47.3 K 2023-04-25 09:53:50,107 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 34fc2da8903b441c9bf18d310e131127, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682416427367 2023-04-25 09:53:50,108 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 96ef4e23c9d146e49d86626d170407c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1682416427701 2023-04-25 09:53:50,109 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b7a7c369f20248f394ca63233aef3bce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682416427976 2023-04-25 09:53:50,109 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 16bbe43a21984646b3584443c056dbff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682416428709 2023-04-25 09:53:50,124 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:50,151 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#38 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:50,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:50,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:50,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:50,160 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:50,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:50,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:50,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:50,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:50,162 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:50,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:50,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:50,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:50,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:50,181 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,181 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:50,181 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:53:50,181 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:50,181 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:50,182 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:50,182 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:50,182 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:50,182 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:50,195 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/7fab608af71c4e81bccd02cdb06fd6d1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7fab608af71c4e81bccd02cdb06fd6d1 2023-04-25 09:53:50,204 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 7fab608af71c4e81bccd02cdb06fd6d1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:50,204 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:50,205 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=12, startTime=1682416429998; duration=0sec 2023-04-25 09:53:50,205 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:50,224 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/817f48cf459c47fb9e3fb8352f5048c3 2023-04-25 09:53:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:50,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:50,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:50,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416490287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,295 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/060b06801e39478f8ef930d231fc03c3 2023-04-25 09:53:50,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416490394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,495 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/776fe84446aa43238d525561b63f2220 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/776fe84446aa43238d525561b63f2220 2023-04-25 09:53:50,505 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 776fe84446aa43238d525561b63f2220(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:50,505 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:50,505 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=12, startTime=1682416429997; duration=0sec 2023-04-25 09:53:50,505 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416490597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,739 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/1950fe90b3cc4f0ebd9f05ef313f1920 2023-04-25 09:53:50,747 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/817f48cf459c47fb9e3fb8352f5048c3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/817f48cf459c47fb9e3fb8352f5048c3 2023-04-25 09:53:50,756 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/817f48cf459c47fb9e3fb8352f5048c3, entries=150, sequenceid=230, filesize=11.8 K 2023-04-25 09:53:50,757 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/060b06801e39478f8ef930d231fc03c3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/060b06801e39478f8ef930d231fc03c3 2023-04-25 09:53:50,764 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/060b06801e39478f8ef930d231fc03c3, entries=150, sequenceid=230, filesize=11.8 K 2023-04-25 09:53:50,765 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/1950fe90b3cc4f0ebd9f05ef313f1920 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/1950fe90b3cc4f0ebd9f05ef313f1920 2023-04-25 09:53:50,771 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/1950fe90b3cc4f0ebd9f05ef313f1920, entries=150, sequenceid=230, filesize=11.8 K 2023-04-25 09:53:50,773 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6d622ed976f9d94b52b9c4f656801b6b in 591ms, sequenceid=230, compaction requested=false 2023-04-25 09:53:50,773 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:50,773 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:50,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,776 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e6cc223[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:50,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:50,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,781 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:50,781 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:50,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:50,785 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:50,785 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:50,785 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:50,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:50,790 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:50,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:50,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:50,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:50,791 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:50,792 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,792 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:50,792 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:50,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:50,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:50,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:50,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:50,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:50,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:50,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:53:50,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:50,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:50,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:50,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:50,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:50,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:50,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:50,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:50,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:50,906 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:50,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:50,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:50,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:50,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,908 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:50,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:50,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:50,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:50,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:50,912 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:50,912 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,912 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:50,913 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:50,913 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:50,918 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,918 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,919 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c731d61[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:50,920 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:50,920 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:50,922 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:50,922 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:50,923 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:50,927 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:50,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:50,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:50,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:50,927 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:50,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:50,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:50,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:50,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:50,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:50,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:50,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:50,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:50,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:50,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:50,936 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:50,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:50,936 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:50,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:50,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:50,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416490951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,036 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:51,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:51,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:51,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:51,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:51,040 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:51,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:51,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:51,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,042 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:51,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:51,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:51,044 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,044 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:51,044 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:51,044 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:51,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ad2721[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:51,051 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:51,051 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:51,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:51,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416491053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:51,056 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:51,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:51,059 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:51,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:51,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:51,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:51,059 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:51,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:51,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:51,068 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:51,068 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:51,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:51,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:51,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:51,068 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,068 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,068 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,068 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,068 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,069 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:51,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:51,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:51,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:51,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:51,174 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:51,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:51,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:51,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:51,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:51,179 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,179 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:51,179 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:51,179 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:51,181 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,181 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,181 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:51,183 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:51,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@114e3268[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:51,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:51,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:51,186 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:51,190 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:51,190 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:51,190 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:51,190 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:51,190 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:51,193 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,193 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,193 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,195 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,200 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,200 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,200 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:51,200 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:51,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:51,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,201 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:51,201 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:51,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416491256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:51,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:51,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:51,308 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:51,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:51,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:51,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,310 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:51,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:51,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:51,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:51,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:51,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:51,315 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59fb87b8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:51,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:51,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/23a9ae22f3cc4143bef1a32f552ef5bc 2023-04-25 09:53:51,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:51,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:51,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:51,322 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:51,331 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:51,331 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:51,331 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:51,331 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:51,331 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:51,333 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,333 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,333 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:51,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,339 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,339 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,339 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:51,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:51,339 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:51,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:51,339 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:51,340 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:51,340 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:51,340 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,340 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,340 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:51,340 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/94fead5d8de44934bcb8f9c65644ecee 2023-04-25 09:53:51,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/c0aaa12957f140b9b9a0bb65c769c0a5 2023-04-25 09:53:51,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/23a9ae22f3cc4143bef1a32f552ef5bc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/23a9ae22f3cc4143bef1a32f552ef5bc 2023-04-25 09:53:51,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/23a9ae22f3cc4143bef1a32f552ef5bc, entries=200, sequenceid=246, filesize=14.1 K 2023-04-25 09:53:51,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/94fead5d8de44934bcb8f9c65644ecee as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/94fead5d8de44934bcb8f9c65644ecee 2023-04-25 09:53:51,440 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:51,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/94fead5d8de44934bcb8f9c65644ecee, entries=150, sequenceid=246, filesize=11.8 K 2023-04-25 09:53:51,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/c0aaa12957f140b9b9a0bb65c769c0a5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c0aaa12957f140b9b9a0bb65c769c0a5 2023-04-25 09:53:51,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:51,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c0aaa12957f140b9b9a0bb65c769c0a5, entries=150, sequenceid=246, filesize=11.8 K 2023-04-25 09:53:51,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:51,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:51,468 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:51,468 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6d622ed976f9d94b52b9c4f656801b6b in 566ms, sequenceid=246, compaction requested=true 2023-04-25 09:53:51,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:51,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:51,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:51,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:51,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:51,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:51,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:51,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:51,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:51,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:51,484 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:51,484 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:51,484 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,484 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:51,484 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:51,484 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:53:51,484 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:51,485 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:51,485 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:51,485 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:51,485 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:51,485 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:51,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:51,486 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:51,486 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:51,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:51,485 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:51,486 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:51,486 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:51,486 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,486 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,486 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/776fe84446aa43238d525561b63f2220, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/817f48cf459c47fb9e3fb8352f5048c3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/23a9ae22f3cc4143bef1a32f552ef5bc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=38.0 K 2023-04-25 09:53:51,486 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fa1109b351f040eb95ec26e46e9d6d7a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/060b06801e39478f8ef930d231fc03c3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/94fead5d8de44934bcb8f9c65644ecee] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=35.7 K 2023-04-25 09:53:51,487 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 776fe84446aa43238d525561b63f2220, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682416428709 2023-04-25 09:53:51,492 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting fa1109b351f040eb95ec26e46e9d6d7a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682416428709 2023-04-25 09:53:51,497 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 817f48cf459c47fb9e3fb8352f5048c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682416429090 2023-04-25 09:53:51,497 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 060b06801e39478f8ef930d231fc03c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682416429090 2023-04-25 09:53:51,498 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 23a9ae22f3cc4143bef1a32f552ef5bc, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416430259 2023-04-25 09:53:51,498 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 94fead5d8de44934bcb8f9c65644ecee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416430259 2023-04-25 09:53:51,513 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:51,516 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:51,539 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/adb70dbe47b64673b9e22cd242e72c3a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/adb70dbe47b64673b9e22cd242e72c3a 2023-04-25 09:53:51,543 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/13eee476edeb4a5b963d05b41055c46c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/13eee476edeb4a5b963d05b41055c46c 2023-04-25 09:53:51,546 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into adb70dbe47b64673b9e22cd242e72c3a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:51,546 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:51,547 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=13, startTime=1682416431484; duration=0sec 2023-04-25 09:53:51,547 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:51,547 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:51,551 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:51,551 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:51,551 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:51,552 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7fab608af71c4e81bccd02cdb06fd6d1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/1950fe90b3cc4f0ebd9f05ef313f1920, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c0aaa12957f140b9b9a0bb65c769c0a5] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=35.7 K 2023-04-25 09:53:51,552 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7fab608af71c4e81bccd02cdb06fd6d1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682416428709 2023-04-25 09:53:51,553 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1950fe90b3cc4f0ebd9f05ef313f1920, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682416429090 2023-04-25 09:53:51,553 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c0aaa12957f140b9b9a0bb65c769c0a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416430259 2023-04-25 09:53:51,556 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 13eee476edeb4a5b963d05b41055c46c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:51,557 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:51,557 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=13, startTime=1682416431484; duration=0sec 2023-04-25 09:53:51,557 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:51,566 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#48 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:51,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:51,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:51,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416491606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416491710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416491854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,858 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:51,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416491873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,877 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:51,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:51,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416491914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:51,916 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/b5e4954a1efc4e858e7544ebaae55a0d 2023-04-25 09:53:51,947 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/82923a1216134f85bcda64e30fc79458 2023-04-25 09:53:51,967 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/01bbde32ca924990bbfb416cc0542670 2023-04-25 09:53:51,975 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/b5e4954a1efc4e858e7544ebaae55a0d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b5e4954a1efc4e858e7544ebaae55a0d 2023-04-25 09:53:51,981 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b5e4954a1efc4e858e7544ebaae55a0d, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 09:53:51,982 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/82923a1216134f85bcda64e30fc79458 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/82923a1216134f85bcda64e30fc79458 2023-04-25 09:53:51,993 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/82923a1216134f85bcda64e30fc79458, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 09:53:51,994 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/01bbde32ca924990bbfb416cc0542670 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/01bbde32ca924990bbfb416cc0542670 2023-04-25 09:53:52,004 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/01bbde32ca924990bbfb416cc0542670, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 09:53:52,005 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6d622ed976f9d94b52b9c4f656801b6b in 521ms, sequenceid=264, compaction requested=false 2023-04-25 09:53:52,005 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:52,005 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:52,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:52,018 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,006 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/501e378ca2024a9da2a7eea2f83d4f92 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/501e378ca2024a9da2a7eea2f83d4f92 2023-04-25 09:53:52,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:52,019 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@db701f3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:52,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:52,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:52,026 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:52,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:52,030 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:52,031 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 501e378ca2024a9da2a7eea2f83d4f92(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:52,031 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:52,031 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=13, startTime=1682416431486; duration=0sec 2023-04-25 09:53:52,031 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:52,034 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:52,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:52,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:52,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:52,034 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:52,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,042 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:52,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:52,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:52,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:52,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:52,047 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:52,047 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:52,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:52,147 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:52,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:52,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:52,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:52,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:52,155 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:52,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:52,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:52,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:52,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:52,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:52,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:52,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:52,160 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:52,161 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:52,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:52,162 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:52,173 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/b0cf086e45db42c29d559ac42908f714 2023-04-25 09:53:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:52,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:52,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416492278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416492279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416492302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,305 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8328 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:53:52,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416492381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416492381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416492583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416492583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,602 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/b61a0a5fc3f9451f9158525013863842 2023-04-25 09:53:52,631 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/9a78f654cb6a47adb457e1df1485fe9d 2023-04-25 09:53:52,642 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/b0cf086e45db42c29d559ac42908f714 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b0cf086e45db42c29d559ac42908f714 2023-04-25 09:53:52,650 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b0cf086e45db42c29d559ac42908f714, entries=150, sequenceid=281, filesize=11.9 K 2023-04-25 09:53:52,651 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/b61a0a5fc3f9451f9158525013863842 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b61a0a5fc3f9451f9158525013863842 2023-04-25 09:53:52,659 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b61a0a5fc3f9451f9158525013863842, entries=150, sequenceid=281, filesize=11.9 K 2023-04-25 09:53:52,661 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/9a78f654cb6a47adb457e1df1485fe9d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9a78f654cb6a47adb457e1df1485fe9d 2023-04-25 09:53:52,672 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9a78f654cb6a47adb457e1df1485fe9d, entries=150, sequenceid=281, filesize=11.9 K 2023-04-25 09:53:52,673 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6d622ed976f9d94b52b9c4f656801b6b in 512ms, sequenceid=281, compaction requested=true 2023-04-25 09:53:52,673 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:52,673 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:52,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:52,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:52,677 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:52,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dab4b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:52,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:52,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,681 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:52,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:52,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:52,682 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:52,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:52,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:52,685 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:52,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:52,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:52,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:52,686 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:52,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:52,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:52,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:52,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:52,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:52,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:52,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:52,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:52,697 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:52,697 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:52,797 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:52,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:52,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:52,819 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:52,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:52,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:52,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:52,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:52,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:52,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:52,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:52,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:52,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:52,825 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:52,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:52,826 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:53:52,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:52,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:52,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:52,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:52,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:52,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:52,840 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/bd7143ce1fad41209181a19ea5161960 2023-04-25 09:53:52,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:52,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:52,897 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/48eb2733e9de452fab93696a5e812fa6 2023-04-25 09:53:52,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416492913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:52,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:52,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416492916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416493018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416493018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416493220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416493221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,327 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/24aab7f48087454f9e9f2140141ca46f 2023-04-25 09:53:53,340 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/bd7143ce1fad41209181a19ea5161960 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/bd7143ce1fad41209181a19ea5161960 2023-04-25 09:53:53,347 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/bd7143ce1fad41209181a19ea5161960, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 09:53:53,349 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/48eb2733e9de452fab93696a5e812fa6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/48eb2733e9de452fab93696a5e812fa6 2023-04-25 09:53:53,355 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/48eb2733e9de452fab93696a5e812fa6, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 09:53:53,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/24aab7f48087454f9e9f2140141ca46f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/24aab7f48087454f9e9f2140141ca46f 2023-04-25 09:53:53,367 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/24aab7f48087454f9e9f2140141ca46f, entries=150, sequenceid=299, filesize=11.9 K 2023-04-25 09:53:53,369 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6d622ed976f9d94b52b9c4f656801b6b in 543ms, sequenceid=299, compaction requested=true 2023-04-25 09:53:53,369 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:53,369 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:53,369 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:53,369 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:53,369 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:53,376 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:53,378 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,378 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63101434[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:53,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:53,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,381 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:53,389 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:53,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:53,396 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:53,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:53,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:53,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:53,396 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:53,398 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,398 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:53,398 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:53,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:53,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,401 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:53,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:53,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:53,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:53,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:53,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:53,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:53,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:53,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:53,404 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:53,404 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:53,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:53,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,504 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:53,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:53,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:53,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:53,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:53,509 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:53,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:53,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:53,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:53,510 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,510 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:53,512 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:53,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:53,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:53,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:53,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:53,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:53,514 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:53,515 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:53,516 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:53,516 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:53,516 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:53,516 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:53,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:53,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:53,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:53,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416493570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416493571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416493677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416493678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416493881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:53,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416493883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:53,949 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/409065d834a247aa942bc671fd9c9360 2023-04-25 09:53:53,974 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/6cfbd1ba4ba7472f862116001c5e343b 2023-04-25 09:53:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416494184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416494184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,393 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/c221d0ba42714accb45159ebb023adb7 2023-04-25 09:53:54,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/409065d834a247aa942bc671fd9c9360 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/409065d834a247aa942bc671fd9c9360 2023-04-25 09:53:54,414 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/409065d834a247aa942bc671fd9c9360, entries=150, sequenceid=313, filesize=11.9 K 2023-04-25 09:53:54,415 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/6cfbd1ba4ba7472f862116001c5e343b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/6cfbd1ba4ba7472f862116001c5e343b 2023-04-25 09:53:54,421 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/6cfbd1ba4ba7472f862116001c5e343b, entries=150, sequenceid=313, filesize=11.9 K 2023-04-25 09:53:54,422 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/c221d0ba42714accb45159ebb023adb7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c221d0ba42714accb45159ebb023adb7 2023-04-25 09:53:54,428 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c221d0ba42714accb45159ebb023adb7, entries=150, sequenceid=313, filesize=11.9 K 2023-04-25 09:53:54,429 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6d622ed976f9d94b52b9c4f656801b6b in 914ms, sequenceid=313, compaction requested=true 2023-04-25 09:53:54,429 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:54,429 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:54,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:54,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665399a9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:54,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:54,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:54,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:54,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:54,445 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:54,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:54,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:54,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:54,445 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:54,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:54,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:54,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:54,457 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:54,457 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,557 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:54,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:54,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:54,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:54,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:54,564 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:54,564 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:54,564 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:54,564 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:54,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:54,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:54,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:54,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:54,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:54,576 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,576 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:54,576 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:53:54,576 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:54,576 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:54,577 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:54,577 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:54,577 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:54,577 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:54,595 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/9ca50bedad754078a84678ebabab0e7b 2023-04-25 09:53:54,622 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/77bfb0d94b874f1fb4c3315477d09030 2023-04-25 09:53:54,637 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/7e10b69ff1fc433ca830190bcc32a7eb 2023-04-25 09:53:54,646 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/9ca50bedad754078a84678ebabab0e7b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9ca50bedad754078a84678ebabab0e7b 2023-04-25 09:53:54,653 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9ca50bedad754078a84678ebabab0e7b, entries=150, sequenceid=331, filesize=11.9 K 2023-04-25 09:53:54,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/77bfb0d94b874f1fb4c3315477d09030 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/77bfb0d94b874f1fb4c3315477d09030 2023-04-25 09:53:54,669 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/77bfb0d94b874f1fb4c3315477d09030, entries=150, sequenceid=331, filesize=11.9 K 2023-04-25 09:53:54,670 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/7e10b69ff1fc433ca830190bcc32a7eb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7e10b69ff1fc433ca830190bcc32a7eb 2023-04-25 09:53:54,679 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7e10b69ff1fc433ca830190bcc32a7eb, entries=150, sequenceid=331, filesize=11.9 K 2023-04-25 09:53:54,680 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 6d622ed976f9d94b52b9c4f656801b6b in 104ms, sequenceid=331, compaction requested=true 2023-04-25 09:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:54,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:54,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53dea409[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:54,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:54,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:54,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:54,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:54,694 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:54,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:54,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:54,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:54,694 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:54,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:54,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:54,697 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:54,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:54,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:54,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:54,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:54,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:54,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:54,704 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:54,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:54,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:54,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:53:54,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:54,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:54,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:54,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:54,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:54,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:54,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/3f3224e1c3db482e882715caa5d5000f 2023-04-25 09:53:54,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:54,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416494783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:54,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416494787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416434747, max timestamp=1682416434784Num uniques 859; 2023-04-25 09:53:54,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:53:54,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416434774, max timestamp=1682416434784Num uniques 859; 2023-04-25 09:53:54,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:54,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416434747, max timestamp=1682416434784Num uniques 859; 2023-04-25 09:53:54,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:53:54,805 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:54,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:54,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/f6f41f82754c4a43907a1da3bfd8e331 2023-04-25 09:53:54,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:54,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:54,813 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:54,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:54,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:54,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:54,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,817 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:54,817 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:54,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:54,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:54,823 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,823 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:54,823 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:54,823 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:54,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:54,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d473aca[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:54,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:54,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:54,836 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:54,843 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:54,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:54,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:54,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:54,843 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:54,848 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,848 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:54,849 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,853 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:54,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:54,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:54,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:54,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:54,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:54,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:54,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:54,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,858 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,858 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,858 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:54,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:54,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:54,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:54,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416494958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:54,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416494958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:54,968 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:54,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:54,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:54,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:54,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,978 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:54,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:54,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:54,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:54,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:54,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:54,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:54,988 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:54,990 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:54,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1538e2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:54,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:54,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:54,997 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:55,003 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:55,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:55,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:55,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:55,003 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:55,007 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,007 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,007 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:55,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,015 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:55,015 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:55,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:55,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416495067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416495076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,116 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:55,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:55,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:55,124 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:55,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:55,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:55,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,126 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,126 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:55,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:55,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,136 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:55,137 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:55,137 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:55,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,139 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f16c58[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:55,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:55,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,145 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,145 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:55,145 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:55,145 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:55,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:55,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:55,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:55,152 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:55,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:55,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:55,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:55,153 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:55,156 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,164 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:55,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,170 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:55,170 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:55,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/dc93cd10c6804dfab9054e7d5269cb72 2023-04-25 09:53:55,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/3f3224e1c3db482e882715caa5d5000f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/3f3224e1c3db482e882715caa5d5000f 2023-04-25 09:53:55,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/3f3224e1c3db482e882715caa5d5000f, entries=250, sequenceid=342, filesize=16.7 K 2023-04-25 09:53:55,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/f6f41f82754c4a43907a1da3bfd8e331 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/f6f41f82754c4a43907a1da3bfd8e331 2023-04-25 09:53:55,270 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:55,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:55,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/f6f41f82754c4a43907a1da3bfd8e331, entries=150, sequenceid=342, filesize=11.9 K 2023-04-25 09:53:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:55,274 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:55,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:55,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:55,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,276 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/dc93cd10c6804dfab9054e7d5269cb72 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/dc93cd10c6804dfab9054e7d5269cb72 2023-04-25 09:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,276 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,276 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:55,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416495269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:55,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:55,278 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,278 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:55,278 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:55,278 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:55,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:55,281 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/dc93cd10c6804dfab9054e7d5269cb72, entries=150, sequenceid=342, filesize=11.9 K 2023-04-25 09:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:55,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for 6d622ed976f9d94b52b9c4f656801b6b in 564ms, sequenceid=342, compaction requested=true 2023-04-25 09:53:55,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:55,283 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 09:53:55,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:55,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:55,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:55,283 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 09:53:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2296a23[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:55,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:55,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,289 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 90544 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 09:53:55,289 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:55,290 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,290 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/13eee476edeb4a5b963d05b41055c46c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b5e4954a1efc4e858e7544ebaae55a0d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b0cf086e45db42c29d559ac42908f714, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/bd7143ce1fad41209181a19ea5161960, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/409065d834a247aa942bc671fd9c9360, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9ca50bedad754078a84678ebabab0e7b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/3f3224e1c3db482e882715caa5d5000f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=88.4 K 2023-04-25 09:53:55,290 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 13eee476edeb4a5b963d05b41055c46c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416430259 2023-04-25 09:53:55,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85664 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 09:53:55,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:55,290 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,291 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/adb70dbe47b64673b9e22cd242e72c3a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/82923a1216134f85bcda64e30fc79458, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b61a0a5fc3f9451f9158525013863842, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/48eb2733e9de452fab93696a5e812fa6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/6cfbd1ba4ba7472f862116001c5e343b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/77bfb0d94b874f1fb4c3315477d09030, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/f6f41f82754c4a43907a1da3bfd8e331] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=83.7 K 2023-04-25 09:53:55,291 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b5e4954a1efc4e858e7544ebaae55a0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682416430938 2023-04-25 09:53:55,291 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:55,291 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting adb70dbe47b64673b9e22cd242e72c3a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416430259 2023-04-25 09:53:55,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:55,291 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b0cf086e45db42c29d559ac42908f714, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682416431589 2023-04-25 09:53:55,292 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 82923a1216134f85bcda64e30fc79458, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682416430938 2023-04-25 09:53:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:55,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-25 09:53:55,292 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b61a0a5fc3f9451f9158525013863842, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682416431589 2023-04-25 09:53:55,292 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting bd7143ce1fad41209181a19ea5161960, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682416432269 2023-04-25 09:53:55,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:55,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:55,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:55,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:55,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:55,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,293 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 48eb2733e9de452fab93696a5e812fa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682416432269 2023-04-25 09:53:55,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:55,293 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 409065d834a247aa942bc671fd9c9360, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682416432894 2023-04-25 09:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,294 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9ca50bedad754078a84678ebabab0e7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416433558 2023-04-25 09:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,294 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6cfbd1ba4ba7472f862116001c5e343b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682416432894 2023-04-25 09:53:55,294 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3f3224e1c3db482e882715caa5d5000f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1682416434706 2023-04-25 09:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,295 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 77bfb0d94b874f1fb4c3315477d09030, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416433558 2023-04-25 09:53:55,295 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f6f41f82754c4a43907a1da3bfd8e331, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1682416434706 2023-04-25 09:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:55,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:55,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:55,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:55,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:55,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:55,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/4e809ed7fa6f40dfbdee085323e5f72b 2023-04-25 09:53:55,307 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,307 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:55,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:55,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,309 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:55,309 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:55,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:55,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416495322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,330 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#68 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:55,337 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#69 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:55,369 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/9216394de5474876b250edb76dca21e3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9216394de5474876b250edb76dca21e3 2023-04-25 09:53:55,375 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/742121b615d54518b16abcb2340d0308 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/742121b615d54518b16abcb2340d0308 2023-04-25 09:53:55,378 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 9216394de5474876b250edb76dca21e3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:55,378 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:55,379 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=9, startTime=1682416435283; duration=0sec 2023-04-25 09:53:55,379 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:55,379 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 09:53:55,382 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85664 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 09:53:55,382 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:55,382 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,383 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/501e378ca2024a9da2a7eea2f83d4f92, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/01bbde32ca924990bbfb416cc0542670, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9a78f654cb6a47adb457e1df1485fe9d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/24aab7f48087454f9e9f2140141ca46f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c221d0ba42714accb45159ebb023adb7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7e10b69ff1fc433ca830190bcc32a7eb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/dc93cd10c6804dfab9054e7d5269cb72] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=83.7 K 2023-04-25 09:53:55,383 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 501e378ca2024a9da2a7eea2f83d4f92, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416430259 2023-04-25 09:53:55,384 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 01bbde32ca924990bbfb416cc0542670, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682416430938 2023-04-25 09:53:55,384 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9a78f654cb6a47adb457e1df1485fe9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682416431589 2023-04-25 09:53:55,385 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 24aab7f48087454f9e9f2140141ca46f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682416432269 2023-04-25 09:53:55,385 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c221d0ba42714accb45159ebb023adb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682416432894 2023-04-25 09:53:55,386 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 742121b615d54518b16abcb2340d0308(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:55,386 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:55,386 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=9, startTime=1682416435283; duration=0sec 2023-04-25 09:53:55,386 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:55,386 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 7e10b69ff1fc433ca830190bcc32a7eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416433558 2023-04-25 09:53:55,386 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting dc93cd10c6804dfab9054e7d5269cb72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1682416434706 2023-04-25 09:53:55,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:55,411 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:55,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:55,416 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:55,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:55,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:55,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:55,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:55,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:55,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:55,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:55,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:55,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:55,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416495433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bc81c6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:55,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:55,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,446 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:55,446 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:55,446 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:55,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:55,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,450 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/c6bca1f4688b484088e1f4039c14cd78 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c6bca1f4688b484088e1f4039c14cd78 2023-04-25 09:53:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:55,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:55,453 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:55,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:55,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:55,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:55,453 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:55,457 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,462 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into c6bca1f4688b484088e1f4039c14cd78(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:55,462 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:55,462 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=9, startTime=1682416435283; duration=0sec 2023-04-25 09:53:55,462 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:55,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:55,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:55,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:55,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:55,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:55,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:55,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:55,580 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:55,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:55,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:55,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,581 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,581 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416495579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,630 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,630 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,631 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,631 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:55,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:55,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:55,634 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:55,634 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,634 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:55,635 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:55,635 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:55,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:55,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:55,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:55,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:55,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:55,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:55,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416495640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71196bdb[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:55,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:55,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,729 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,745 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,745 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1c203ec5f1ef4eac9c30f872b8d9451a 2023-04-25 09:53:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:55,757 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:55,757 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:55,757 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:55,757 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:55,757 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:55,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/60099b717f8441408ca83ded291f80fd 2023-04-25 09:53:55,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/4e809ed7fa6f40dfbdee085323e5f72b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4e809ed7fa6f40dfbdee085323e5f72b 2023-04-25 09:53:55,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4e809ed7fa6f40dfbdee085323e5f72b, entries=200, sequenceid=379, filesize=14.3 K 2023-04-25 09:53:55,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1c203ec5f1ef4eac9c30f872b8d9451a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1c203ec5f1ef4eac9c30f872b8d9451a 2023-04-25 09:53:55,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:55,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,780 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:55,782 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:55,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1c203ec5f1ef4eac9c30f872b8d9451a, entries=150, sequenceid=379, filesize=11.9 K 2023-04-25 09:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/60099b717f8441408ca83ded291f80fd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/60099b717f8441408ca83ded291f80fd 2023-04-25 09:53:55,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/60099b717f8441408ca83ded291f80fd, entries=150, sequenceid=379, filesize=11.9 K 2023-04-25 09:53:55,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 6d622ed976f9d94b52b9c4f656801b6b in 500ms, sequenceid=379, compaction requested=false 2023-04-25 09:53:55,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:55,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,830 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:55,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,830 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:55,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:55,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:55,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:53:55,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:55,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:55,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:55,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:55,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:55,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:55,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/18ba01b2394b4084b30d8d824e919be4 2023-04-25 09:53:55,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416435906, max timestamp=1682416435911Num uniques 859; 2023-04-25 09:53:55,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:55,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416435906, max timestamp=1682416435911Num uniques 859; 2023-04-25 09:53:55,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:55,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416435906, max timestamp=1682416435910Num uniques 859; 2023-04-25 09:53:55,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:55,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/b5da0a9d70e34b469d360904c5b2515c 2023-04-25 09:53:55,930 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:55,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:55,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:55,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:55,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:55,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:55,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:55,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:55,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/2c7a296062e14fa28fd654d669f59c23 2023-04-25 09:53:55,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:55,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:55,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416495954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416495955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:55,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:55,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:55,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416495957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/18ba01b2394b4084b30d8d824e919be4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/18ba01b2394b4084b30d8d824e919be4 2023-04-25 09:53:55,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/18ba01b2394b4084b30d8d824e919be4, entries=150, sequenceid=392, filesize=11.9 K 2023-04-25 09:53:55,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/b5da0a9d70e34b469d360904c5b2515c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b5da0a9d70e34b469d360904c5b2515c 2023-04-25 09:53:55,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:55,977 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,977 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:55,977 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:55,977 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:55,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b5da0a9d70e34b469d360904c5b2515c, entries=150, sequenceid=392, filesize=11.9 K 2023-04-25 09:53:55,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/2c7a296062e14fa28fd654d669f59c23 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2c7a296062e14fa28fd654d669f59c23 2023-04-25 09:53:55,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2c7a296062e14fa28fd654d669f59c23, entries=150, sequenceid=392, filesize=11.9 K 2023-04-25 09:53:55,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6d622ed976f9d94b52b9c4f656801b6b in 104ms, sequenceid=392, compaction requested=true 2023-04-25 09:53:55,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:55,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:55,988 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:55,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:55,988 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:55,988 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:55,989 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:55,990 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:55,990 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:55,990 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,990 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:55,990 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9216394de5474876b250edb76dca21e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4e809ed7fa6f40dfbdee085323e5f72b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/18ba01b2394b4084b30d8d824e919be4] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=38.8 K 2023-04-25 09:53:55,990 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:55,990 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/742121b615d54518b16abcb2340d0308, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1c203ec5f1ef4eac9c30f872b8d9451a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b5da0a9d70e34b469d360904c5b2515c] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=36.4 K 2023-04-25 09:53:55,990 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9216394de5474876b250edb76dca21e3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1682416434706 2023-04-25 09:53:55,991 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 742121b615d54518b16abcb2340d0308, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1682416434706 2023-04-25 09:53:55,991 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4e809ed7fa6f40dfbdee085323e5f72b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682416434932 2023-04-25 09:53:55,992 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1c203ec5f1ef4eac9c30f872b8d9451a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682416434941 2023-04-25 09:53:55,992 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 18ba01b2394b4084b30d8d824e919be4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1682416435314 2023-04-25 09:53:55,992 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b5da0a9d70e34b469d360904c5b2515c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1682416435314 2023-04-25 09:53:55,999 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea54a0c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,007 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:56,009 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:56,019 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/46f0af77225b4681b3f5a84abeaf84c4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/46f0af77225b4681b3f5a84abeaf84c4 2023-04-25 09:53:56,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,027 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/1b891e2e89564cc488589199c12d07e3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1b891e2e89564cc488589199c12d07e3 2023-04-25 09:53:56,027 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 46f0af77225b4681b3f5a84abeaf84c4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:56,027 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:56,027 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=13, startTime=1682416435988; duration=0sec 2023-04-25 09:53:56,027 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:56,027 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:56,029 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:56,030 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:56,030 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,030 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c6bca1f4688b484088e1f4039c14cd78, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/60099b717f8441408ca83ded291f80fd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2c7a296062e14fa28fd654d669f59c23] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=36.4 K 2023-04-25 09:53:56,030 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c6bca1f4688b484088e1f4039c14cd78, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1682416434706 2023-04-25 09:53:56,031 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 60099b717f8441408ca83ded291f80fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682416434941 2023-04-25 09:53:56,031 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2c7a296062e14fa28fd654d669f59c23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1682416435314 2023-04-25 09:53:56,034 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 1b891e2e89564cc488589199c12d07e3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:56,034 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:56,034 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=13, startTime=1682416435988; duration=0sec 2023-04-25 09:53:56,034 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:56,045 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#77 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,060 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-25 09:53:56,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416435926, max timestamp=1682416436059Num uniques 1018; 2023-04-25 09:53:56,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 09:53:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:56,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:56,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:56,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,064 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:56,064 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,066 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,066 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,083 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/90bf26b4494e4f12af7c9c97c0e054d3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/90bf26b4494e4f12af7c9c97c0e054d3 2023-04-25 09:53:56,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416496084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,091 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 90bf26b4494e4f12af7c9c97c0e054d3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:56,091 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:56,091 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=13, startTime=1682416435988; duration=0sec 2023-04-25 09:53:56,091 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:56,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416496087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416496088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36802 deadline: 1682416496088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1 2023-04-25 09:53:56,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/8dd33937b9364188860d9a163b44d0bb 2023-04-25 09:53:56,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:56,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:56,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/a5dbb79e619b40af85ff83583a1b3f1c 2023-04-25 09:53:56,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,161 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:56,161 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:56,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:56,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1 2023-04-25 09:53:56,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1, entries=250, sequenceid=433, filesize=16.7 K 2023-04-25 09:53:56,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/8dd33937b9364188860d9a163b44d0bb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/8dd33937b9364188860d9a163b44d0bb 2023-04-25 09:53:56,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/8dd33937b9364188860d9a163b44d0bb, entries=150, sequenceid=433, filesize=11.9 K 2023-04-25 09:53:56,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/a5dbb79e619b40af85ff83583a1b3f1c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a5dbb79e619b40af85ff83583a1b3f1c 2023-04-25 09:53:56,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a5dbb79e619b40af85ff83583a1b3f1c, entries=150, sequenceid=433, filesize=11.9 K 2023-04-25 09:53:56,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~109.58 KB/112210, heapSize ~312.08 KB/319568, currentSize=73.80 KB/75570 for 6d622ed976f9d94b52b9c4f656801b6b in 134ms, sequenceid=433, compaction requested=false 2023-04-25 09:53:56,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:56,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:56,203 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:53:56,205 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:56,205 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:56,205 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:56,205 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:56,205 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:56,205 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:56,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416496218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416496222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416496224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,261 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:56,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:56,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:56,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:56,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:56,265 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:56,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:56,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:56,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:56,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:56,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:56,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:56,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:56,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:56,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:56,274 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17145b89[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,278 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,281 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,284 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,284 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:56,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:56,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,320 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:56,320 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:56,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416496324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416496329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416496330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,420 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:56,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:56,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:56,424 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:56,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:56,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:56,425 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,425 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:56,432 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,436 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75c45a1c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,443 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,447 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,447 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:56,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:56,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:56,461 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:56,461 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:56,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416496527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416496532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416496532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,561 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:56,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:56,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:56,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:56,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:56,565 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:56,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:56,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:56,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:56,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:56,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:56,571 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,574 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,575 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb9a0cc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,580 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,581 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,581 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,584 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,585 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:56,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:56,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,591 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:56,591 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:56,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:56,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/6b8fba28694a43bf847f0e31975572bf 2023-04-25 09:53:56,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/fc3d11427aee4c31b36caf024f503d5d 2023-04-25 09:53:56,691 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:56,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:56,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:56,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:56,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:56,701 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:56,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:56,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:56,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,704 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:56,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:56,706 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,706 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:56,706 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:56,706 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:56,710 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb8761d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,715 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,715 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,716 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,716 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,717 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,717 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,720 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,720 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,720 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,720 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,720 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,723 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,723 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,723 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,727 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:56,727 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:56,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,727 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,727 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:56,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,827 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:56,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:56,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:56,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:56,832 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:56,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:56,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:56,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416496832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,835 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:56,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:56,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:56,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416496836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416496837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:56,840 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,840 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:56,840 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:56,840 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,842 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec5baf0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,845 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,845 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,850 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,850 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,852 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,855 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,856 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:56,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:56,863 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,863 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,863 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:56,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:56,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:56,964 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:56,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:56,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:56,978 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:56,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:56,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:56,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,981 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:56,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:56,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:56,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:56,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:56,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:56,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:56,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:56,986 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5986158e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:56,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:56,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,990 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:56,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:56,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:56,995 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:56,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:56,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:56,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:56,996 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:56,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:56,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:56,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:57,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:57,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:57,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,004 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:57,004 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:57,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416497098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/a83cec13323c4bac95e5b62e197d2ac1 2023-04-25 09:53:57,104 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:57,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:57,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:57,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:57,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:57,111 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:57,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:57,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:57,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:57,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:57,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:57,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/6b8fba28694a43bf847f0e31975572bf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6b8fba28694a43bf847f0e31975572bf 2023-04-25 09:53:57,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:57,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:57,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6b8fba28694a43bf847f0e31975572bf, entries=200, sequenceid=451, filesize=14.3 K 2023-04-25 09:53:57,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/fc3d11427aee4c31b36caf024f503d5d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fc3d11427aee4c31b36caf024f503d5d 2023-04-25 09:53:57,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fc3d11427aee4c31b36caf024f503d5d, entries=150, sequenceid=451, filesize=11.9 K 2023-04-25 09:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:57,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:57,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:57,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/a83cec13323c4bac95e5b62e197d2ac1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a83cec13323c4bac95e5b62e197d2ac1 2023-04-25 09:53:57,137 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:57,137 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:57,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,144 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a83cec13323c4bac95e5b62e197d2ac1, entries=150, sequenceid=451, filesize=11.9 K 2023-04-25 09:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:57,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6d622ed976f9d94b52b9c4f656801b6b in 946ms, sequenceid=451, compaction requested=true 2023-04-25 09:53:57,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:57,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:57,150 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:57,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:57,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbb61b2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:57,150 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:57,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:57,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:57,152 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:57,152 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:57,152 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:57,152 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1b891e2e89564cc488589199c12d07e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6b8fba28694a43bf847f0e31975572bf] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=43.7 K 2023-04-25 09:53:57,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:57,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:57,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:57,153 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1b891e2e89564cc488589199c12d07e3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1682416435314 2023-04-25 09:53:57,153 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:57,153 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:57,154 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6e4a90e4f69d4a469fc4cc7aca1ec4d1, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682416435926 2023-04-25 09:53:57,154 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:57,154 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/46f0af77225b4681b3f5a84abeaf84c4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/8dd33937b9364188860d9a163b44d0bb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fc3d11427aee4c31b36caf024f503d5d] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=36.5 K 2023-04-25 09:53:57,154 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6b8fba28694a43bf847f0e31975572bf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682416436086 2023-04-25 09:53:57,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:57,154 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 46f0af77225b4681b3f5a84abeaf84c4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1682416435314 2023-04-25 09:53:57,155 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8dd33937b9364188860d9a163b44d0bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682416435926 2023-04-25 09:53:57,155 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:57,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:57,156 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting fc3d11427aee4c31b36caf024f503d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682416436201 2023-04-25 09:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:57,161 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:57,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:57,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:57,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:57,162 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:57,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:57,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:57,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:57,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,174 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#84 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,178 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#85 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:57,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:57,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:57,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:57,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:57,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:57,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:57,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:57,219 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/586120cfef79468baa2032e1d3f4fd1c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/586120cfef79468baa2032e1d3f4fd1c 2023-04-25 09:53:57,219 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1de4d7430cbe4d54bc3bd670c542171c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1de4d7430cbe4d54bc3bd670c542171c 2023-04-25 09:53:57,234 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 1de4d7430cbe4d54bc3bd670c542171c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:57,234 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:57,234 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=13, startTime=1682416437150; duration=0sec 2023-04-25 09:53:57,234 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:57,234 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:53:57,236 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 586120cfef79468baa2032e1d3f4fd1c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:57,236 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:57,236 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:53:57,236 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:57,236 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=13, startTime=1682416437150; duration=0sec 2023-04-25 09:53:57,236 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:57,236 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:57,236 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/90bf26b4494e4f12af7c9c97c0e054d3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a5dbb79e619b40af85ff83583a1b3f1c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a83cec13323c4bac95e5b62e197d2ac1] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=36.5 K 2023-04-25 09:53:57,237 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 90bf26b4494e4f12af7c9c97c0e054d3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1682416435314 2023-04-25 09:53:57,237 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a5dbb79e619b40af85ff83583a1b3f1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1682416435926 2023-04-25 09:53:57,240 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a83cec13323c4bac95e5b62e197d2ac1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682416436201 2023-04-25 09:53:57,268 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:57,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:57,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:57,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:57,296 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:57,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:57,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:57,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:57,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:57,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,302 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,302 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:57,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:57,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:57,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:57,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:57,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:57,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:57,304 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:57,304 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:57,304 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:53:57,304 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:57,304 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:57,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:57,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:57,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:57,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:57,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:57,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416497378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416497381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416497381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416497483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416497483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416497483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416497686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416497686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36842 deadline: 1682416497687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,714 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/acd83434476142ff8b39e7331102bf18 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/acd83434476142ff8b39e7331102bf18 2023-04-25 09:53:57,721 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into acd83434476142ff8b39e7331102bf18(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:57,721 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:57,721 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=13, startTime=1682416437151; duration=0sec 2023-04-25 09:53:57,721 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:57,728 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/68b472d66aa44e2c91e30bdf3992f1c1 2023-04-25 09:53:57,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36782 deadline: 1682416497989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416497990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416497990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,173 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/3971833659cf4574933d2fbcd509f54c 2023-04-25 09:53:58,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416498496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:58,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416498496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416498496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,598 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/393212c5c30948639e89a64973f2ebb4 2023-04-25 09:53:58,604 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/68b472d66aa44e2c91e30bdf3992f1c1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/68b472d66aa44e2c91e30bdf3992f1c1 2023-04-25 09:53:58,616 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/68b472d66aa44e2c91e30bdf3992f1c1, entries=150, sequenceid=468, filesize=11.9 K 2023-04-25 09:53:58,617 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/3971833659cf4574933d2fbcd509f54c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/3971833659cf4574933d2fbcd509f54c 2023-04-25 09:53:58,622 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/3971833659cf4574933d2fbcd509f54c, entries=150, sequenceid=468, filesize=11.9 K 2023-04-25 09:53:58,623 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/393212c5c30948639e89a64973f2ebb4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/393212c5c30948639e89a64973f2ebb4 2023-04-25 09:53:58,628 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/393212c5c30948639e89a64973f2ebb4, entries=150, sequenceid=468, filesize=11.9 K 2023-04-25 09:53:58,629 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for 6d622ed976f9d94b52b9c4f656801b6b in 1325ms, sequenceid=468, compaction requested=false 2023-04-25 09:53:58,629 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:58,629 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:58,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:58,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23cbec46[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:58,634 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:58,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,636 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:58,636 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,636 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:58,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:58,640 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:58,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:58,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:58,646 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:58,646 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:58,646 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:58,646 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:58,646 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:58,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:58,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:58,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,656 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,656 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:58,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:58,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:58,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:58,656 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:58,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:58,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:58,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:58,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:58,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:58,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:58,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:58,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:58,761 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:58,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:58,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:58,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:58,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:58,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:58,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:58,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:58,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:58,766 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:58,766 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:58,766 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:58,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:58,778 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/a57dcd1f72fe4e6385065a30d4b65dfe 2023-04-25 09:53:58,801 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/ae9153510da44b9ebcc0c8f9633d39b8 2023-04-25 09:53:58,832 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/ce096422ed404207a53decdd96a81fa2 2023-04-25 09:53:58,838 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/a57dcd1f72fe4e6385065a30d4b65dfe as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a57dcd1f72fe4e6385065a30d4b65dfe 2023-04-25 09:53:58,846 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a57dcd1f72fe4e6385065a30d4b65dfe, entries=150, sequenceid=488, filesize=11.9 K 2023-04-25 09:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/ae9153510da44b9ebcc0c8f9633d39b8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/ae9153510da44b9ebcc0c8f9633d39b8 2023-04-25 09:53:58,855 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/ae9153510da44b9ebcc0c8f9633d39b8, entries=150, sequenceid=488, filesize=11.9 K 2023-04-25 09:53:58,855 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/ce096422ed404207a53decdd96a81fa2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/ce096422ed404207a53decdd96a81fa2 2023-04-25 09:53:58,861 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/ce096422ed404207a53decdd96a81fa2, entries=150, sequenceid=488, filesize=11.9 K 2023-04-25 09:53:58,862 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 6d622ed976f9d94b52b9c4f656801b6b in 96ms, sequenceid=488, compaction requested=true 2023-04-25 09:53:58,862 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:58,862 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:58,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:58,870 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fad0f5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:58,875 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:58,875 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:58,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:58,879 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:58,883 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:58,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:58,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:58,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:58,883 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:58,885 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,885 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,885 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:58,887 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,890 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:58,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:58,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:58,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,898 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:58,898 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:58,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:58,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:58,898 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:58,898 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:58,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:58,998 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:58,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:59,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:59,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:59,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:59,003 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:59,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:59,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:59,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:59,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,005 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:59,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:59,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:59,007 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:59,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:59,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:59,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:59,009 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:59,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69b98f01[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:59,012 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:59,012 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,013 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:59,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:59,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:59,018 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:59,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:59,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:59,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:59,018 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:59,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:59,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:59,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,023 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:59,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:59,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:59,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:59,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:59,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:59,026 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:59,026 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:59,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,027 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,027 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:59,027 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,126 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:59,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:59,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:59,130 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:59,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:59,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:59,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:59,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,131 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,132 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:59,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:59,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:59,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:59,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:59,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:59,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:59,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:59,138 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,138 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:59,138 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:59,138 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:53:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:53:59,142 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,143 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:53:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:59,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:53:59,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690fd575[Count = 0] remaining members to acquire global barrier 2023-04-25 09:53:59,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:53:59,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,147 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:53:59,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:53:59,150 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:53:59,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/6a0188586ee4467e95ac981d89b8a3d9 2023-04-25 09:53:59,156 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:53:59,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:53:59,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:53:59,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:53:59,156 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:53:59,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:53:59,162 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:53:59,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:59,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:53:59,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:59,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:53:59,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:53:59,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:53:59,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,169 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:53:59,170 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:53:59,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:53:59,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:53:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:53:59,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:53:59,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416439188, max timestamp=1682416439203Num uniques 859; 2023-04-25 09:53:59,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416439188, max timestamp=1682416439203Num uniques 859; 2023-04-25 09:53:59,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416439188, max timestamp=1682416439203Num uniques 859; 2023-04-25 09:53:59,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:59,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:59,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:59,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/04a62d0c3d8b449faea366dc2c8b019f 2023-04-25 09:53:59,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/6a7aba7a4abc44b9a8b44e694fc1a1d9 2023-04-25 09:53:59,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/6a0188586ee4467e95ac981d89b8a3d9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6a0188586ee4467e95ac981d89b8a3d9 2023-04-25 09:53:59,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6a0188586ee4467e95ac981d89b8a3d9, entries=50, sequenceid=499, filesize=7.1 K 2023-04-25 09:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/04a62d0c3d8b449faea366dc2c8b019f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/04a62d0c3d8b449faea366dc2c8b019f 2023-04-25 09:53:59,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:53:59,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:53:59,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:53:59,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/04a62d0c3d8b449faea366dc2c8b019f, entries=50, sequenceid=499, filesize=7.1 K 2023-04-25 09:53:59,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/6a7aba7a4abc44b9a8b44e694fc1a1d9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/6a7aba7a4abc44b9a8b44e694fc1a1d9 2023-04-25 09:53:59,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/6a7aba7a4abc44b9a8b44e694fc1a1d9, entries=50, sequenceid=499, filesize=7.1 K 2023-04-25 09:53:59,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:53:59,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:53:59,285 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:53:59,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:59,285 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6d622ed976f9d94b52b9c4f656801b6b in 149ms, sequenceid=499, compaction requested=true 2023-04-25 09:53:59,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:53:59,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:53:59,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:59,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,286 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:53:59,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:59,286 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:53:59,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:59,287 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:59,288 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44812 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:53:59,288 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:53:59,288 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,288 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/586120cfef79468baa2032e1d3f4fd1c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/68b472d66aa44e2c91e30bdf3992f1c1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a57dcd1f72fe4e6385065a30d4b65dfe, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6a0188586ee4467e95ac981d89b8a3d9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=43.8 K 2023-04-25 09:53:59,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:53:59,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:53:59,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:53:59,289 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44812 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:53:59,289 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:53:59,289 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 586120cfef79468baa2032e1d3f4fd1c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682416436201 2023-04-25 09:53:59,289 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,290 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1de4d7430cbe4d54bc3bd670c542171c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/3971833659cf4574933d2fbcd509f54c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/ae9153510da44b9ebcc0c8f9633d39b8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/04a62d0c3d8b449faea366dc2c8b019f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=43.8 K 2023-04-25 09:53:59,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:53:59,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:53:59,290 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 68b472d66aa44e2c91e30bdf3992f1c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1682416436215 2023-04-25 09:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:53:59,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:53:59,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:53:59,291 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a57dcd1f72fe4e6385065a30d4b65dfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682416437358 2023-04-25 09:53:59,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:53:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:53:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:53:59,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:53:59,291 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1de4d7430cbe4d54bc3bd670c542171c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682416436201 2023-04-25 09:53:59,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416439252, max timestamp=1682416439289Num uniques 1018; 2023-04-25 09:53:59,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:53:59,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416439252, max timestamp=1682416439289Num uniques 1018; 2023-04-25 09:53:59,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:53:59,293 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6a0188586ee4467e95ac981d89b8a3d9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416439135 2023-04-25 09:53:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:53:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:53:59,294 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,294 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:53:59,294 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3971833659cf4574933d2fbcd509f54c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1682416436215 2023-04-25 09:53:59,294 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=62.62 KB heapSize=169.03 KB 2023-04-25 09:53:59,295 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ae9153510da44b9ebcc0c8f9633d39b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682416437358 2023-04-25 09:53:59,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416439265, max timestamp=1682416439289Num uniques 1018; 2023-04-25 09:53:59,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 09:53:59,296 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 04a62d0c3d8b449faea366dc2c8b019f, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416439135 2023-04-25 09:53:59,298 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:53:59,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:59,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:53:59,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:59,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:53:59,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:53:59,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:53:59,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:53:59,330 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:59,352 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#98 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:59,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416439365, max timestamp=1682416439390Num uniques 859; 2023-04-25 09:53:59,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:59,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416439365, max timestamp=1682416439390Num uniques 859; 2023-04-25 09:53:59,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:59,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416439365, max timestamp=1682416439390Num uniques 859; 2023-04-25 09:53:59,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:53:59,465 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/8381fde6471e4b4a8d79f6c72df297cc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/8381fde6471e4b4a8d79f6c72df297cc 2023-04-25 09:53:59,473 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/24272a7f2b76471085f3f1c6ba6ad3cb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/24272a7f2b76471085f3f1c6ba6ad3cb 2023-04-25 09:53:59,473 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 8381fde6471e4b4a8d79f6c72df297cc(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:59,473 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:59,473 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=12, startTime=1682416439286; duration=0sec 2023-04-25 09:53:59,473 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:53:59,473 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:53:59,478 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44812 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:53:59,478 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:53:59,478 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:53:59,478 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/acd83434476142ff8b39e7331102bf18, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/393212c5c30948639e89a64973f2ebb4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/ce096422ed404207a53decdd96a81fa2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/6a7aba7a4abc44b9a8b44e694fc1a1d9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=43.8 K 2023-04-25 09:53:59,480 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting acd83434476142ff8b39e7331102bf18, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682416436201 2023-04-25 09:53:59,481 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 393212c5c30948639e89a64973f2ebb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1682416436215 2023-04-25 09:53:59,482 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ce096422ed404207a53decdd96a81fa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1682416437358 2023-04-25 09:53:59,483 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 24272a7f2b76471085f3f1c6ba6ad3cb(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:59,483 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:59,483 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=12, startTime=1682416439286; duration=0sec 2023-04-25 09:53:59,483 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:59,483 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6a7aba7a4abc44b9a8b44e694fc1a1d9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416439135 2023-04-25 09:53:59,501 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:53:59,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416439464, max timestamp=1682416439504Num uniques 1018; 2023-04-25 09:53:59,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:53:59,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416439464, max timestamp=1682416439504Num uniques 1018; 2023-04-25 09:53:59,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:53:59,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416439464, max timestamp=1682416439504Num uniques 1018; 2023-04-25 09:53:59,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:53:59,522 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/0c6cca26c904482c9e7b686b0fd0aa7a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0c6cca26c904482c9e7b686b0fd0aa7a 2023-04-25 09:53:59,527 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 0c6cca26c904482c9e7b686b0fd0aa7a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:53:59,527 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:53:59,527 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=12, startTime=1682416439287; duration=0sec 2023-04-25 09:53:59,527 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:53:59,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1682416439516, max timestamp=1682416439537Num uniques 1077; 2023-04-25 09:53:59,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=21842, suffix heap size=139976,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=477 2023-04-25 09:53:59,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1682416439516, max timestamp=1682416439537Num uniques 1077; 2023-04-25 09:53:59,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1682416439516, max timestamp=1682416439537Num uniques 1077; 2023-04-25 09:53:59,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=21842, suffix heap size=139976,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=477 2023-04-25 09:53:59,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=21842, suffix heap size=139976,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=477 2023-04-25 09:53:59,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416499587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416499587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416499588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416499589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416499695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416499696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416499696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416499697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,768 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/56b875e72aae42b283620cc7a3226ff6 2023-04-25 09:53:59,786 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/bada3b1c5bf7446898b879f2536cce02 2023-04-25 09:53:59,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416499898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416499901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416499901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:53:59,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:53:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416499902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,197 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/f6d356adf1ca4031896d86a6d236a614 2023-04-25 09:54:00,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416500202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,204 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/56b875e72aae42b283620cc7a3226ff6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/56b875e72aae42b283620cc7a3226ff6 2023-04-25 09:54:00,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416500204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416500205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416500205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,210 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/56b875e72aae42b283620cc7a3226ff6, entries=150, sequenceid=537, filesize=11.9 K 2023-04-25 09:54:00,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/bada3b1c5bf7446898b879f2536cce02 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/bada3b1c5bf7446898b879f2536cce02 2023-04-25 09:54:00,216 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/bada3b1c5bf7446898b879f2536cce02, entries=150, sequenceid=537, filesize=11.9 K 2023-04-25 09:54:00,217 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/f6d356adf1ca4031896d86a6d236a614 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/f6d356adf1ca4031896d86a6d236a614 2023-04-25 09:54:00,222 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/f6d356adf1ca4031896d86a6d236a614, entries=150, sequenceid=537, filesize=11.9 K 2023-04-25 09:54:00,222 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~31.31 KB/32060, heapSize ~75.50 KB/77312, currentSize=167.72 KB/171750 for 6d622ed976f9d94b52b9c4f656801b6b in 928ms, sequenceid=537, compaction requested=false 2023-04-25 09:54:00,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:00,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:00,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:00,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:00,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:00,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:00,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155abdce[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:00,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:00,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:00,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:00,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:00,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:00,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:00,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:00,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:00,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:00,232 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:00,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:00,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:00,233 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:00,233 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:00,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:00,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:00,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:00,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:00,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:00,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:00,239 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:00,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:00,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:00,340 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:00,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:00,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:00,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:00,344 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:00,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:00,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:00,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:00,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:00,345 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:00,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:00,346 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,346 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:00,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:00,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:00,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:00,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:00,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:00,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:00,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:54:00,348 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=167.72 KB heapSize=454.78 KB 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:54:00,348 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:00,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:54:00,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416500709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416500710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416500711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416500712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,757 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/57adf9390290467f9ff131780b13c68b 2023-04-25 09:54:00,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416500814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416500814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:00,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416500814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416501017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416501017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416501018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,179 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/7aeb588fa8a34d389e7b04b0f2320d26 2023-04-25 09:54:01,197 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/46844d5cbe134134b483f13a44b7729e 2023-04-25 09:54:01,203 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/57adf9390290467f9ff131780b13c68b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/57adf9390290467f9ff131780b13c68b 2023-04-25 09:54:01,210 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/57adf9390290467f9ff131780b13c68b, entries=150, sequenceid=610, filesize=11.9 K 2023-04-25 09:54:01,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/7aeb588fa8a34d389e7b04b0f2320d26 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/7aeb588fa8a34d389e7b04b0f2320d26 2023-04-25 09:54:01,215 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/7aeb588fa8a34d389e7b04b0f2320d26, entries=150, sequenceid=610, filesize=11.9 K 2023-04-25 09:54:01,216 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/46844d5cbe134134b483f13a44b7729e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/46844d5cbe134134b483f13a44b7729e 2023-04-25 09:54:01,221 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/46844d5cbe134134b483f13a44b7729e, entries=150, sequenceid=610, filesize=11.9 K 2023-04-25 09:54:01,222 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~454.73 KB/465648, currentSize=20.13 KB/20610 for 6d622ed976f9d94b52b9c4f656801b6b in 874ms, sequenceid=610, compaction requested=true 2023-04-25 09:54:01,222 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:01,222 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:01,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:01,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4371bae8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:01,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:01,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:01,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:01,243 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:01,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:01,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:01,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:01,243 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:01,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:01,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:01,251 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:01,252 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:01,252 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:01,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:01,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:54:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:54:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:54:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:01,339 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/a0c279df89a341188fd68d5fe3a80125 2023-04-25 09:54:01,352 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:01,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:01,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416501357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416501358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416501357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/310ed99a137c40d4b4e6a834dc43c0c9 2023-04-25 09:54:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:01,368 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:01,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:01,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:01,369 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,369 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,370 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,370 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,370 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,370 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:01,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:01,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:01,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:54:01,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:54:01,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:01,374 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,376 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:01,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6695f400[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:01,378 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:01,378 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,379 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,379 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:01,379 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:01,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:01,379 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:01,381 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:01,384 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:01,384 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:01,384 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:01,384 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:01,384 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:01,386 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,386 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,386 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,387 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,389 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/5be81f79b27d45d3b0cf9ef492faa5ec 2023-04-25 09:54:01,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:01,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:01,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:01,397 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:01,397 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/a0c279df89a341188fd68d5fe3a80125 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a0c279df89a341188fd68d5fe3a80125 2023-04-25 09:54:01,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a0c279df89a341188fd68d5fe3a80125, entries=150, sequenceid=622, filesize=11.9 K 2023-04-25 09:54:01,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/310ed99a137c40d4b4e6a834dc43c0c9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/310ed99a137c40d4b4e6a834dc43c0c9 2023-04-25 09:54:01,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/310ed99a137c40d4b4e6a834dc43c0c9, entries=150, sequenceid=622, filesize=11.9 K 2023-04-25 09:54:01,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/5be81f79b27d45d3b0cf9ef492faa5ec as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5be81f79b27d45d3b0cf9ef492faa5ec 2023-04-25 09:54:01,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5be81f79b27d45d3b0cf9ef492faa5ec, entries=150, sequenceid=622, filesize=11.9 K 2023-04-25 09:54:01,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6d622ed976f9d94b52b9c4f656801b6b in 117ms, sequenceid=622, compaction requested=true 2023-04-25 09:54:01,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:01,445 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:01,445 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:01,445 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:01,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:01,448 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:01,448 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:54:01,448 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,448 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/8381fde6471e4b4a8d79f6c72df297cc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/56b875e72aae42b283620cc7a3226ff6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/57adf9390290467f9ff131780b13c68b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a0c279df89a341188fd68d5fe3a80125] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=48.7 K 2023-04-25 09:54:01,448 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8381fde6471e4b4a8d79f6c72df297cc, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416437372 2023-04-25 09:54:01,450 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:01,450 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:54:01,450 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,450 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/24272a7f2b76471085f3f1c6ba6ad3cb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/bada3b1c5bf7446898b879f2536cce02, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/7aeb588fa8a34d389e7b04b0f2320d26, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/310ed99a137c40d4b4e6a834dc43c0c9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=48.7 K 2023-04-25 09:54:01,451 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 24272a7f2b76471085f3f1c6ba6ad3cb, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416437372 2023-04-25 09:54:01,451 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bada3b1c5bf7446898b879f2536cce02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682416439265 2023-04-25 09:54:01,452 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7aeb588fa8a34d389e7b04b0f2320d26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682416439552 2023-04-25 09:54:01,452 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 56b875e72aae42b283620cc7a3226ff6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682416439265 2023-04-25 09:54:01,453 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 310ed99a137c40d4b4e6a834dc43c0c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682416440710 2023-04-25 09:54:01,453 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 57adf9390290467f9ff131780b13c68b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682416439552 2023-04-25 09:54:01,453 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a0c279df89a341188fd68d5fe3a80125, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682416440710 2023-04-25 09:54:01,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416441342, max timestamp=1682416441467Num uniques 859; 2023-04-25 09:54:01,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:01,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416441342, max timestamp=1682416441467Num uniques 859; 2023-04-25 09:54:01,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:01,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:01,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=64.85 KB heapSize=184.09 KB 2023-04-25 09:54:01,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416441342, max timestamp=1682416441470Num uniques 859; 2023-04-25 09:54:01,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:01,475 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#108 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:01,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:54:01,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:01,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:54:01,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:01,477 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:54:01,477 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:01,491 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:01,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:01,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:01,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416441488, max timestamp=1682416441512Num uniques 859; 2023-04-25 09:54:01,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:54:01,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416441488, max timestamp=1682416441512Num uniques 859; 2023-04-25 09:54:01,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416441488, max timestamp=1682416441512Num uniques 859; 2023-04-25 09:54:01,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:54:01,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:54:01,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:01,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:01,518 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:01,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:01,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:01,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,520 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,520 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,520 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:01,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:01,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416501543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:01,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:01,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416501546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416501549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:01,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:01,566 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,566 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:54:01,567 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:54:01,567 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:01,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:01,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:01,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:01,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/caa817cc788749bb84ea63649e27c0e3 2023-04-25 09:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:01,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a621da[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:01,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:01,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,575 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,575 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:01,575 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:01,575 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:01,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:01,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:01,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:01,583 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:01,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:01,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:01,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:01,583 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:01,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,586 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:01,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,590 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:01,590 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416501650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416501653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416501655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,690 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:01,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:01,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:01,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:01,696 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:01,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:01,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:01,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,697 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,700 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:01,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:01,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:01,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:01,701 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,701 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:54:01,701 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:54:01,701 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:01,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:01,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:01,703 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,703 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,703 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:01,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31d246ed[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:01,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:01,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:01,707 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:01,707 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,707 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:01,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:01,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:01,711 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:01,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:01,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:01,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:01,711 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:01,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36798 deadline: 1682416501714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,715 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:01,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:01,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:01,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,719 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:01,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:01,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,820 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:01,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:01,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:01,824 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:01,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:01,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:01,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,826 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,826 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:01,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:01,830 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,830 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:54:01,830 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:54:01,830 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:01,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:01,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fba1cc5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:01,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:01,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,836 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:01,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:01,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:01,836 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,836 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:01,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:01,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:01,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:01,840 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:01,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:01,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:01,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:01,840 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:01,842 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,842 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,842 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,842 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,850 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:01,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,850 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:01,850 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:01,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416501853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416501856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:01,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416501860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,950 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:01,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:01,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:01,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:01,954 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:01,954 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,954 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:01,954 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:01,954 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,959 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,959 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,959 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,959 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:01,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,960 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/0c19797e997c4a74a5a365080342a5bf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0c19797e997c4a74a5a365080342a5bf 2023-04-25 09:54:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:01,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:01,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:01,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,967 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 0c19797e997c4a74a5a365080342a5bf(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:01,967 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:01,967 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=12, startTime=1682416441445; duration=0sec 2023-04-25 09:54:01,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. started... 2023-04-25 09:54:01,968 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. as already flushing 2023-04-25 09:54:01,968 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:01,969 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:01,969 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,970 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,972 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:01,972 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,973 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:01,973 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0c6cca26c904482c9e7b686b0fd0aa7a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/f6d356adf1ca4031896d86a6d236a614, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/46844d5cbe134134b483f13a44b7729e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5be81f79b27d45d3b0cf9ef492faa5ec] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=48.7 K 2023-04-25 09:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3646c2c6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:01,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:01,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,973 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/fb2036a138d34130a9ffcf7ac317c2c0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fb2036a138d34130a9ffcf7ac317c2c0 2023-04-25 09:54:01,974 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0c6cca26c904482c9e7b686b0fd0aa7a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416437372 2023-04-25 09:54:01,974 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f6d356adf1ca4031896d86a6d236a614, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1682416439265 2023-04-25 09:54:01,975 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:01,975 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 46844d5cbe134134b483f13a44b7729e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1682416439552 2023-04-25 09:54:01,975 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,975 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:01,976 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5be81f79b27d45d3b0cf9ef492faa5ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682416440710 2023-04-25 09:54:01,978 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:01,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,980 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into fb2036a138d34130a9ffcf7ac317c2c0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:01,980 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:01,980 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=12, startTime=1682416441445; duration=0sec 2023-04-25 09:54:01,980 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:01,984 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:01,985 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:01,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:01,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:01,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:01,985 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:01,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:01,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:01,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:01,987 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:01,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:01,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:01,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:01,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:01,992 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:01,992 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:01,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:01,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:02,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/e95213587b9347eb9996c1e5b749db4e 2023-04-25 09:54:02,002 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:60014 2023-04-25 09:54:02,002 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,003 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:60014 2023-04-25 09:54:02,003 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,007 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:60014 2023-04-25 09:54:02,007 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,008 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:60014 2023-04-25 09:54:02,008 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,017 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/2d1a9ad9eed1495d8038dcc8d5ac6a8d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2d1a9ad9eed1495d8038dcc8d5ac6a8d 2023-04-25 09:54:02,022 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 2d1a9ad9eed1495d8038dcc8d5ac6a8d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:02,023 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:02,023 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=12, startTime=1682416441446; duration=0sec 2023-04-25 09:54:02,023 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:02,092 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:02,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:02,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:02,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36782 deadline: 1682416502160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:02,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:02,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36842 deadline: 1682416502160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:02,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:02,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36802 deadline: 1682416502163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:02,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:02,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36840 deadline: 1682416502353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:02,354 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18377 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:02,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=645 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/9ceddbce7cf24871b638557585621e93 2023-04-25 09:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/caa817cc788749bb84ea63649e27c0e3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/caa817cc788749bb84ea63649e27c0e3 2023-04-25 09:54:02,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/caa817cc788749bb84ea63649e27c0e3, entries=200, sequenceid=645, filesize=14.3 K 2023-04-25 09:54:02,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/e95213587b9347eb9996c1e5b749db4e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/e95213587b9347eb9996c1e5b749db4e 2023-04-25 09:54:02,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/e95213587b9347eb9996c1e5b749db4e, entries=150, sequenceid=645, filesize=11.9 K 2023-04-25 09:54:02,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/9ceddbce7cf24871b638557585621e93 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9ceddbce7cf24871b638557585621e93 2023-04-25 09:54:02,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9ceddbce7cf24871b638557585621e93, entries=150, sequenceid=645, filesize=11.9 K 2023-04-25 09:54:02,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for 6d622ed976f9d94b52b9c4f656801b6b in 979ms, sequenceid=645, compaction requested=false 2023-04-25 09:54:02,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:02,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:02,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-25 09:54:02,666 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:60014 2023-04-25 09:54:02,666 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:54:02,667 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:60014 2023-04-25 09:54:02,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:02,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:54:02,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:02,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:54:02,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:02,667 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,669 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:60014 2023-04-25 09:54:02,669 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:02,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/60773bf291d8486e8b7deb50c203d2c7 2023-04-25 09:54:03,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/a2db51b948c441dfbbdce77cefe2a164 2023-04-25 09:54:03,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/e09f757de1d5475d91a8c62ec580fe16 2023-04-25 09:54:03,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/60773bf291d8486e8b7deb50c203d2c7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60773bf291d8486e8b7deb50c203d2c7 2023-04-25 09:54:03,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60773bf291d8486e8b7deb50c203d2c7, entries=150, sequenceid=685, filesize=11.9 K 2023-04-25 09:54:03,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/a2db51b948c441dfbbdce77cefe2a164 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/a2db51b948c441dfbbdce77cefe2a164 2023-04-25 09:54:03,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/a2db51b948c441dfbbdce77cefe2a164, entries=150, sequenceid=685, filesize=11.9 K 2023-04-25 09:54:03,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/e09f757de1d5475d91a8c62ec580fe16 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e09f757de1d5475d91a8c62ec580fe16 2023-04-25 09:54:03,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e09f757de1d5475d91a8c62ec580fe16, entries=150, sequenceid=685, filesize=11.9 K 2023-04-25 09:54:03,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=6.71 KB/6870 for 6d622ed976f9d94b52b9c4f656801b6b in 885ms, sequenceid=685, compaction requested=true 2023-04-25 09:54:03,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:03,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:03,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:03,552 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:03,552 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:03,552 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:03,553 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:03,553 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:03,553 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/A is initiating minor compaction (all files) 2023-04-25 09:54:03,553 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/B is initiating minor compaction (all files) 2023-04-25 09:54:03,553 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/A in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:03,553 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/B in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:03,553 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fb2036a138d34130a9ffcf7ac317c2c0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/caa817cc788749bb84ea63649e27c0e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60773bf291d8486e8b7deb50c203d2c7] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=39.3 K 2023-04-25 09:54:03,553 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0c19797e997c4a74a5a365080342a5bf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/e95213587b9347eb9996c1e5b749db4e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/a2db51b948c441dfbbdce77cefe2a164] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=36.9 K 2023-04-25 09:54:03,554 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting fb2036a138d34130a9ffcf7ac317c2c0, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682416440710 2023-04-25 09:54:03,554 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0c19797e997c4a74a5a365080342a5bf, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682416440710 2023-04-25 09:54:03,554 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting caa817cc788749bb84ea63649e27c0e3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1682416441355 2023-04-25 09:54:03,554 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e95213587b9347eb9996c1e5b749db4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1682416441355 2023-04-25 09:54:03,554 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 60773bf291d8486e8b7deb50c203d2c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1682416441542 2023-04-25 09:54:03,554 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a2db51b948c441dfbbdce77cefe2a164, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1682416441542 2023-04-25 09:54:03,565 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#B#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:03,567 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#A#compaction#118 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:03,580 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/1a64be2302164706a5f850be93f16553 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1a64be2302164706a5f850be93f16553 2023-04-25 09:54:03,585 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/A of 6d622ed976f9d94b52b9c4f656801b6b into 1a64be2302164706a5f850be93f16553(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:03,585 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:03,585 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/A, priority=13, startTime=1682416443552; duration=0sec 2023-04-25 09:54:03,585 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:03,585 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:03,586 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:03,586 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6d622ed976f9d94b52b9c4f656801b6b/C is initiating minor compaction (all files) 2023-04-25 09:54:03,587 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6d622ed976f9d94b52b9c4f656801b6b/C in TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:03,587 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2d1a9ad9eed1495d8038dcc8d5ac6a8d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9ceddbce7cf24871b638557585621e93, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e09f757de1d5475d91a8c62ec580fe16] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp, totalSize=36.9 K 2023-04-25 09:54:03,587 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2d1a9ad9eed1495d8038dcc8d5ac6a8d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682416440710 2023-04-25 09:54:03,587 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9ceddbce7cf24871b638557585621e93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=645, earliestPutTs=1682416441355 2023-04-25 09:54:03,588 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e09f757de1d5475d91a8c62ec580fe16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1682416441542 2023-04-25 09:54:03,596 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6d622ed976f9d94b52b9c4f656801b6b#C#compaction#119 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:03,730 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:60014 2023-04-25 09:54:03,730 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:03,976 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/1b645ef52b5a48a1af1b6507bb48266f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1b645ef52b5a48a1af1b6507bb48266f 2023-04-25 09:54:03,981 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/B of 6d622ed976f9d94b52b9c4f656801b6b into 1b645ef52b5a48a1af1b6507bb48266f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:03,981 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:03,981 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/B, priority=13, startTime=1682416443552; duration=0sec 2023-04-25 09:54:03,981 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:04,005 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/5afe68810c674ccc86b17f78e8b003d7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5afe68810c674ccc86b17f78e8b003d7 2023-04-25 09:54:04,010 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6d622ed976f9d94b52b9c4f656801b6b/C of 6d622ed976f9d94b52b9c4f656801b6b into 5afe68810c674ccc86b17f78e8b003d7(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:04,010 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:04,010 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b., storeName=6d622ed976f9d94b52b9c4f656801b6b/C, priority=13, startTime=1682416443552; duration=0sec 2023-04-25 09:54:04,010 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:12,397 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:60014 2023-04-25 09:54:12,397 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:12,397 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 250 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4300 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4250 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1902 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5705 rows 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1879 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5636 rows 2023-04-25 09:54:12,398 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:54:12,398 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:60014 2023-04-25 09:54:12,398 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:12,405 INFO [Listener at localhost/33705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 09:54:12,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 09:54:12,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:12,423 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416452423"}]},"ts":"1682416452423"} 2023-04-25 09:54:12,425 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 09:54:12,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 09:54:12,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 09:54:12,602 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 09:54:12,604 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, UNASSIGN}] 2023-04-25 09:54:12,606 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, UNASSIGN 2023-04-25 09:54:12,607 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6d622ed976f9d94b52b9c4f656801b6b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:12,607 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416452607"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416452607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416452607"}]},"ts":"1682416452607"} 2023-04-25 09:54:12,609 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:54:12,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 09:54:12,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:12,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6d622ed976f9d94b52b9c4f656801b6b, disabling compactions & flushes 2023-04-25 09:54:12,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. after waiting 0 ms 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:12,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6d622ed976f9d94b52b9c4f656801b6b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=A 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=B 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6d622ed976f9d94b52b9c4f656801b6b, store=C 2023-04-25 09:54:12,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 09:54:13,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/cdf52bfa4244485e90ee9b4ccfdc097d 2023-04-25 09:54:13,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 09:54:13,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/b4968ad5cbcd4340a7fa589642f77f93 2023-04-25 09:54:14,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/e28ebd997a534e48a612c6fb98064b06 2023-04-25 09:54:14,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/A/cdf52bfa4244485e90ee9b4ccfdc097d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cdf52bfa4244485e90ee9b4ccfdc097d 2023-04-25 09:54:14,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cdf52bfa4244485e90ee9b4ccfdc097d, entries=100, sequenceid=694, filesize=9.5 K 2023-04-25 09:54:14,011 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/B/b4968ad5cbcd4340a7fa589642f77f93 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b4968ad5cbcd4340a7fa589642f77f93 2023-04-25 09:54:14,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b4968ad5cbcd4340a7fa589642f77f93, entries=100, sequenceid=694, filesize=9.5 K 2023-04-25 09:54:14,016 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/.tmp/C/e28ebd997a534e48a612c6fb98064b06 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e28ebd997a534e48a612c6fb98064b06 2023-04-25 09:54:14,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e28ebd997a534e48a612c6fb98064b06, entries=100, sequenceid=694, filesize=9.5 K 2023-04-25 09:54:14,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 6d622ed976f9d94b52b9c4f656801b6b in 1252ms, sequenceid=694, compaction requested=false 2023-04-25 09:54:14,040 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fc1c20d5f6a044a9b9a3af9457388329, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4011e030d0744b169570be7927b3f212, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0f12414a98aa43ae8176fc4f49af6166, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/401df10a3760489aba74156476d7ae90, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/648018cbdbfc4684a0f543d458c3d7f7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cbe2e5436aa0493cbcdfef9cd092f8ca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/852d58246241480692d5d892afcf9544, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/d7224f3b66ed4ab59ef8340ee1d2fc47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/61b72f31b68c416abfdad9320374d1ef, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60c9e7a28d384c55be35531ec663fc08, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/85b82c74a2614c36b931bc2dc4091164, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/776fe84446aa43238d525561b63f2220, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0e375922354947dbbc87c1ce2506319f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/817f48cf459c47fb9e3fb8352f5048c3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/23a9ae22f3cc4143bef1a32f552ef5bc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/13eee476edeb4a5b963d05b41055c46c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b5e4954a1efc4e858e7544ebaae55a0d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b0cf086e45db42c29d559ac42908f714, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/bd7143ce1fad41209181a19ea5161960, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/409065d834a247aa942bc671fd9c9360, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9ca50bedad754078a84678ebabab0e7b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/3f3224e1c3db482e882715caa5d5000f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9216394de5474876b250edb76dca21e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4e809ed7fa6f40dfbdee085323e5f72b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1b891e2e89564cc488589199c12d07e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/18ba01b2394b4084b30d8d824e919be4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6b8fba28694a43bf847f0e31975572bf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/586120cfef79468baa2032e1d3f4fd1c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/68b472d66aa44e2c91e30bdf3992f1c1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a57dcd1f72fe4e6385065a30d4b65dfe, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/8381fde6471e4b4a8d79f6c72df297cc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6a0188586ee4467e95ac981d89b8a3d9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/56b875e72aae42b283620cc7a3226ff6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/57adf9390290467f9ff131780b13c68b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fb2036a138d34130a9ffcf7ac317c2c0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a0c279df89a341188fd68d5fe3a80125, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/caa817cc788749bb84ea63649e27c0e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60773bf291d8486e8b7deb50c203d2c7] to archive 2023-04-25 09:54:14,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:54:14,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fc1c20d5f6a044a9b9a3af9457388329 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fc1c20d5f6a044a9b9a3af9457388329 2023-04-25 09:54:14,047 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4011e030d0744b169570be7927b3f212 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4011e030d0744b169570be7927b3f212 2023-04-25 09:54:14,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0f12414a98aa43ae8176fc4f49af6166 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0f12414a98aa43ae8176fc4f49af6166 2023-04-25 09:54:14,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/401df10a3760489aba74156476d7ae90 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/401df10a3760489aba74156476d7ae90 2023-04-25 09:54:14,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/648018cbdbfc4684a0f543d458c3d7f7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/648018cbdbfc4684a0f543d458c3d7f7 2023-04-25 09:54:14,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cbe2e5436aa0493cbcdfef9cd092f8ca to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cbe2e5436aa0493cbcdfef9cd092f8ca 2023-04-25 09:54:14,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/852d58246241480692d5d892afcf9544 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/852d58246241480692d5d892afcf9544 2023-04-25 09:54:14,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/d7224f3b66ed4ab59ef8340ee1d2fc47 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/d7224f3b66ed4ab59ef8340ee1d2fc47 2023-04-25 09:54:14,057 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/61b72f31b68c416abfdad9320374d1ef to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/61b72f31b68c416abfdad9320374d1ef 2023-04-25 09:54:14,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60c9e7a28d384c55be35531ec663fc08 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60c9e7a28d384c55be35531ec663fc08 2023-04-25 09:54:14,059 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/85b82c74a2614c36b931bc2dc4091164 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/85b82c74a2614c36b931bc2dc4091164 2023-04-25 09:54:14,060 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/776fe84446aa43238d525561b63f2220 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/776fe84446aa43238d525561b63f2220 2023-04-25 09:54:14,061 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0e375922354947dbbc87c1ce2506319f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/0e375922354947dbbc87c1ce2506319f 2023-04-25 09:54:14,062 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/817f48cf459c47fb9e3fb8352f5048c3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/817f48cf459c47fb9e3fb8352f5048c3 2023-04-25 09:54:14,064 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/23a9ae22f3cc4143bef1a32f552ef5bc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/23a9ae22f3cc4143bef1a32f552ef5bc 2023-04-25 09:54:14,065 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/13eee476edeb4a5b963d05b41055c46c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/13eee476edeb4a5b963d05b41055c46c 2023-04-25 09:54:14,066 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b5e4954a1efc4e858e7544ebaae55a0d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b5e4954a1efc4e858e7544ebaae55a0d 2023-04-25 09:54:14,068 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b0cf086e45db42c29d559ac42908f714 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/b0cf086e45db42c29d559ac42908f714 2023-04-25 09:54:14,069 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/bd7143ce1fad41209181a19ea5161960 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/bd7143ce1fad41209181a19ea5161960 2023-04-25 09:54:14,071 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/409065d834a247aa942bc671fd9c9360 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/409065d834a247aa942bc671fd9c9360 2023-04-25 09:54:14,072 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9ca50bedad754078a84678ebabab0e7b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9ca50bedad754078a84678ebabab0e7b 2023-04-25 09:54:14,073 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/3f3224e1c3db482e882715caa5d5000f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/3f3224e1c3db482e882715caa5d5000f 2023-04-25 09:54:14,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9216394de5474876b250edb76dca21e3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/9216394de5474876b250edb76dca21e3 2023-04-25 09:54:14,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4e809ed7fa6f40dfbdee085323e5f72b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/4e809ed7fa6f40dfbdee085323e5f72b 2023-04-25 09:54:14,076 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1b891e2e89564cc488589199c12d07e3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1b891e2e89564cc488589199c12d07e3 2023-04-25 09:54:14,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/18ba01b2394b4084b30d8d824e919be4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/18ba01b2394b4084b30d8d824e919be4 2023-04-25 09:54:14,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6e4a90e4f69d4a469fc4cc7aca1ec4d1 2023-04-25 09:54:14,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6b8fba28694a43bf847f0e31975572bf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6b8fba28694a43bf847f0e31975572bf 2023-04-25 09:54:14,080 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/586120cfef79468baa2032e1d3f4fd1c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/586120cfef79468baa2032e1d3f4fd1c 2023-04-25 09:54:14,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/68b472d66aa44e2c91e30bdf3992f1c1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/68b472d66aa44e2c91e30bdf3992f1c1 2023-04-25 09:54:14,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a57dcd1f72fe4e6385065a30d4b65dfe to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a57dcd1f72fe4e6385065a30d4b65dfe 2023-04-25 09:54:14,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/8381fde6471e4b4a8d79f6c72df297cc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/8381fde6471e4b4a8d79f6c72df297cc 2023-04-25 09:54:14,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6a0188586ee4467e95ac981d89b8a3d9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/6a0188586ee4467e95ac981d89b8a3d9 2023-04-25 09:54:14,086 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/56b875e72aae42b283620cc7a3226ff6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/56b875e72aae42b283620cc7a3226ff6 2023-04-25 09:54:14,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/57adf9390290467f9ff131780b13c68b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/57adf9390290467f9ff131780b13c68b 2023-04-25 09:54:14,088 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fb2036a138d34130a9ffcf7ac317c2c0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/fb2036a138d34130a9ffcf7ac317c2c0 2023-04-25 09:54:14,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a0c279df89a341188fd68d5fe3a80125 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/a0c279df89a341188fd68d5fe3a80125 2023-04-25 09:54:14,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/caa817cc788749bb84ea63649e27c0e3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/caa817cc788749bb84ea63649e27c0e3 2023-04-25 09:54:14,091 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60773bf291d8486e8b7deb50c203d2c7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/60773bf291d8486e8b7deb50c203d2c7 2023-04-25 09:54:14,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2d5ff2eaaa424de0aa8938e2b6a59c17, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/dc9a99cbc973449c97fd6e4d899d9539, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/015c83af25fd4fd9a64e04fdf56b66db, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1406e0bb5af6441a88aa03a10305b840, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1ab32767dc1b4b7eb27d443a98695964, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/23ea49e422b44ba3857397c1e2f8c33b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/826271dca455474aada2acae3ecc708b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/d5d97552dc5f4629983806943640effe, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/5c50fc50352b4481a30dab60271e9c09, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0dcddf69e47440bc9500e89a0b79de35, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/063ab73ce6f546f69c667c89bea7aee5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fa1109b351f040eb95ec26e46e9d6d7a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2113dc558b64475686c139afad4fa6fb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/060b06801e39478f8ef930d231fc03c3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/adb70dbe47b64673b9e22cd242e72c3a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/94fead5d8de44934bcb8f9c65644ecee, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/82923a1216134f85bcda64e30fc79458, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b61a0a5fc3f9451f9158525013863842, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/48eb2733e9de452fab93696a5e812fa6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/6cfbd1ba4ba7472f862116001c5e343b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/77bfb0d94b874f1fb4c3315477d09030, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/742121b615d54518b16abcb2340d0308, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/f6f41f82754c4a43907a1da3bfd8e331, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1c203ec5f1ef4eac9c30f872b8d9451a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/46f0af77225b4681b3f5a84abeaf84c4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b5da0a9d70e34b469d360904c5b2515c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/8dd33937b9364188860d9a163b44d0bb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1de4d7430cbe4d54bc3bd670c542171c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fc3d11427aee4c31b36caf024f503d5d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/3971833659cf4574933d2fbcd509f54c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/ae9153510da44b9ebcc0c8f9633d39b8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/24272a7f2b76471085f3f1c6ba6ad3cb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/04a62d0c3d8b449faea366dc2c8b019f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/bada3b1c5bf7446898b879f2536cce02, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/7aeb588fa8a34d389e7b04b0f2320d26, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0c19797e997c4a74a5a365080342a5bf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/310ed99a137c40d4b4e6a834dc43c0c9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/e95213587b9347eb9996c1e5b749db4e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/a2db51b948c441dfbbdce77cefe2a164] to archive 2023-04-25 09:54:14,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:54:14,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2d5ff2eaaa424de0aa8938e2b6a59c17 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2d5ff2eaaa424de0aa8938e2b6a59c17 2023-04-25 09:54:14,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/dc9a99cbc973449c97fd6e4d899d9539 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/dc9a99cbc973449c97fd6e4d899d9539 2023-04-25 09:54:14,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/015c83af25fd4fd9a64e04fdf56b66db to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/015c83af25fd4fd9a64e04fdf56b66db 2023-04-25 09:54:14,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1406e0bb5af6441a88aa03a10305b840 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1406e0bb5af6441a88aa03a10305b840 2023-04-25 09:54:14,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1ab32767dc1b4b7eb27d443a98695964 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1ab32767dc1b4b7eb27d443a98695964 2023-04-25 09:54:14,122 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/23ea49e422b44ba3857397c1e2f8c33b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/23ea49e422b44ba3857397c1e2f8c33b 2023-04-25 09:54:14,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/826271dca455474aada2acae3ecc708b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/826271dca455474aada2acae3ecc708b 2023-04-25 09:54:14,124 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/d5d97552dc5f4629983806943640effe to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/d5d97552dc5f4629983806943640effe 2023-04-25 09:54:14,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/5c50fc50352b4481a30dab60271e9c09 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/5c50fc50352b4481a30dab60271e9c09 2023-04-25 09:54:14,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0dcddf69e47440bc9500e89a0b79de35 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0dcddf69e47440bc9500e89a0b79de35 2023-04-25 09:54:14,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/063ab73ce6f546f69c667c89bea7aee5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/063ab73ce6f546f69c667c89bea7aee5 2023-04-25 09:54:14,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fa1109b351f040eb95ec26e46e9d6d7a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fa1109b351f040eb95ec26e46e9d6d7a 2023-04-25 09:54:14,129 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2113dc558b64475686c139afad4fa6fb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/2113dc558b64475686c139afad4fa6fb 2023-04-25 09:54:14,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/060b06801e39478f8ef930d231fc03c3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/060b06801e39478f8ef930d231fc03c3 2023-04-25 09:54:14,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/adb70dbe47b64673b9e22cd242e72c3a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/adb70dbe47b64673b9e22cd242e72c3a 2023-04-25 09:54:14,132 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/94fead5d8de44934bcb8f9c65644ecee to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/94fead5d8de44934bcb8f9c65644ecee 2023-04-25 09:54:14,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/82923a1216134f85bcda64e30fc79458 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/82923a1216134f85bcda64e30fc79458 2023-04-25 09:54:14,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b61a0a5fc3f9451f9158525013863842 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b61a0a5fc3f9451f9158525013863842 2023-04-25 09:54:14,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/48eb2733e9de452fab93696a5e812fa6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/48eb2733e9de452fab93696a5e812fa6 2023-04-25 09:54:14,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/6cfbd1ba4ba7472f862116001c5e343b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/6cfbd1ba4ba7472f862116001c5e343b 2023-04-25 09:54:14,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/77bfb0d94b874f1fb4c3315477d09030 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/77bfb0d94b874f1fb4c3315477d09030 2023-04-25 09:54:14,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/742121b615d54518b16abcb2340d0308 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/742121b615d54518b16abcb2340d0308 2023-04-25 09:54:14,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/f6f41f82754c4a43907a1da3bfd8e331 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/f6f41f82754c4a43907a1da3bfd8e331 2023-04-25 09:54:14,140 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1c203ec5f1ef4eac9c30f872b8d9451a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1c203ec5f1ef4eac9c30f872b8d9451a 2023-04-25 09:54:14,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/46f0af77225b4681b3f5a84abeaf84c4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/46f0af77225b4681b3f5a84abeaf84c4 2023-04-25 09:54:14,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b5da0a9d70e34b469d360904c5b2515c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b5da0a9d70e34b469d360904c5b2515c 2023-04-25 09:54:14,144 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/8dd33937b9364188860d9a163b44d0bb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/8dd33937b9364188860d9a163b44d0bb 2023-04-25 09:54:14,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1de4d7430cbe4d54bc3bd670c542171c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1de4d7430cbe4d54bc3bd670c542171c 2023-04-25 09:54:14,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fc3d11427aee4c31b36caf024f503d5d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/fc3d11427aee4c31b36caf024f503d5d 2023-04-25 09:54:14,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/3971833659cf4574933d2fbcd509f54c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/3971833659cf4574933d2fbcd509f54c 2023-04-25 09:54:14,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/ae9153510da44b9ebcc0c8f9633d39b8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/ae9153510da44b9ebcc0c8f9633d39b8 2023-04-25 09:54:14,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/24272a7f2b76471085f3f1c6ba6ad3cb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/24272a7f2b76471085f3f1c6ba6ad3cb 2023-04-25 09:54:14,150 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/04a62d0c3d8b449faea366dc2c8b019f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/04a62d0c3d8b449faea366dc2c8b019f 2023-04-25 09:54:14,151 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/bada3b1c5bf7446898b879f2536cce02 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/bada3b1c5bf7446898b879f2536cce02 2023-04-25 09:54:14,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/7aeb588fa8a34d389e7b04b0f2320d26 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/7aeb588fa8a34d389e7b04b0f2320d26 2023-04-25 09:54:14,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0c19797e997c4a74a5a365080342a5bf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/0c19797e997c4a74a5a365080342a5bf 2023-04-25 09:54:14,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/310ed99a137c40d4b4e6a834dc43c0c9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/310ed99a137c40d4b4e6a834dc43c0c9 2023-04-25 09:54:14,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/e95213587b9347eb9996c1e5b749db4e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/e95213587b9347eb9996c1e5b749db4e 2023-04-25 09:54:14,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/a2db51b948c441dfbbdce77cefe2a164 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/a2db51b948c441dfbbdce77cefe2a164 2023-04-25 09:54:14,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e7f80737e8ae4d40ac9bed23c51273da, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/232da319eb2f49dc9a6946351f3d607b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/54ffc61544674c71a32eba54f5ebde47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0276ad2b88c1432d8ec0dc54c25cc3c3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5d0f727d3263451b954450ce993fe8b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2a4f5e7f0fb94b3092ba43a4ed733e1a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7adc6288e57f4ca89caa92f7a9869150, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/34fc2da8903b441c9bf18d310e131127, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/369c647950964c7a9730f3e8efdae3b2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/96ef4e23c9d146e49d86626d170407c7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/b7a7c369f20248f394ca63233aef3bce, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7fab608af71c4e81bccd02cdb06fd6d1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/16bbe43a21984646b3584443c056dbff, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/1950fe90b3cc4f0ebd9f05ef313f1920, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/501e378ca2024a9da2a7eea2f83d4f92, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c0aaa12957f140b9b9a0bb65c769c0a5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/01bbde32ca924990bbfb416cc0542670, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9a78f654cb6a47adb457e1df1485fe9d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/24aab7f48087454f9e9f2140141ca46f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c221d0ba42714accb45159ebb023adb7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7e10b69ff1fc433ca830190bcc32a7eb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c6bca1f4688b484088e1f4039c14cd78, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/dc93cd10c6804dfab9054e7d5269cb72, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/60099b717f8441408ca83ded291f80fd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/90bf26b4494e4f12af7c9c97c0e054d3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2c7a296062e14fa28fd654d669f59c23, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a5dbb79e619b40af85ff83583a1b3f1c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/acd83434476142ff8b39e7331102bf18, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a83cec13323c4bac95e5b62e197d2ac1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/393212c5c30948639e89a64973f2ebb4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/ce096422ed404207a53decdd96a81fa2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0c6cca26c904482c9e7b686b0fd0aa7a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/6a7aba7a4abc44b9a8b44e694fc1a1d9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/f6d356adf1ca4031896d86a6d236a614, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/46844d5cbe134134b483f13a44b7729e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2d1a9ad9eed1495d8038dcc8d5ac6a8d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5be81f79b27d45d3b0cf9ef492faa5ec, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9ceddbce7cf24871b638557585621e93, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e09f757de1d5475d91a8c62ec580fe16] to archive 2023-04-25 09:54:14,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:54:14,163 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e7f80737e8ae4d40ac9bed23c51273da to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e7f80737e8ae4d40ac9bed23c51273da 2023-04-25 09:54:14,164 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/232da319eb2f49dc9a6946351f3d607b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/232da319eb2f49dc9a6946351f3d607b 2023-04-25 09:54:14,165 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/54ffc61544674c71a32eba54f5ebde47 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/54ffc61544674c71a32eba54f5ebde47 2023-04-25 09:54:14,166 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0276ad2b88c1432d8ec0dc54c25cc3c3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0276ad2b88c1432d8ec0dc54c25cc3c3 2023-04-25 09:54:14,167 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5d0f727d3263451b954450ce993fe8b9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5d0f727d3263451b954450ce993fe8b9 2023-04-25 09:54:14,168 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2a4f5e7f0fb94b3092ba43a4ed733e1a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2a4f5e7f0fb94b3092ba43a4ed733e1a 2023-04-25 09:54:14,169 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7adc6288e57f4ca89caa92f7a9869150 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7adc6288e57f4ca89caa92f7a9869150 2023-04-25 09:54:14,170 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/34fc2da8903b441c9bf18d310e131127 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/34fc2da8903b441c9bf18d310e131127 2023-04-25 09:54:14,171 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/369c647950964c7a9730f3e8efdae3b2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/369c647950964c7a9730f3e8efdae3b2 2023-04-25 09:54:14,172 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/96ef4e23c9d146e49d86626d170407c7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/96ef4e23c9d146e49d86626d170407c7 2023-04-25 09:54:14,173 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/b7a7c369f20248f394ca63233aef3bce to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/b7a7c369f20248f394ca63233aef3bce 2023-04-25 09:54:14,174 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7fab608af71c4e81bccd02cdb06fd6d1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7fab608af71c4e81bccd02cdb06fd6d1 2023-04-25 09:54:14,175 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/16bbe43a21984646b3584443c056dbff to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/16bbe43a21984646b3584443c056dbff 2023-04-25 09:54:14,176 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/1950fe90b3cc4f0ebd9f05ef313f1920 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/1950fe90b3cc4f0ebd9f05ef313f1920 2023-04-25 09:54:14,177 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/501e378ca2024a9da2a7eea2f83d4f92 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/501e378ca2024a9da2a7eea2f83d4f92 2023-04-25 09:54:14,178 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c0aaa12957f140b9b9a0bb65c769c0a5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c0aaa12957f140b9b9a0bb65c769c0a5 2023-04-25 09:54:14,179 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/01bbde32ca924990bbfb416cc0542670 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/01bbde32ca924990bbfb416cc0542670 2023-04-25 09:54:14,180 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9a78f654cb6a47adb457e1df1485fe9d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9a78f654cb6a47adb457e1df1485fe9d 2023-04-25 09:54:14,181 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/24aab7f48087454f9e9f2140141ca46f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/24aab7f48087454f9e9f2140141ca46f 2023-04-25 09:54:14,182 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c221d0ba42714accb45159ebb023adb7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c221d0ba42714accb45159ebb023adb7 2023-04-25 09:54:14,183 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7e10b69ff1fc433ca830190bcc32a7eb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/7e10b69ff1fc433ca830190bcc32a7eb 2023-04-25 09:54:14,184 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c6bca1f4688b484088e1f4039c14cd78 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/c6bca1f4688b484088e1f4039c14cd78 2023-04-25 09:54:14,185 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/dc93cd10c6804dfab9054e7d5269cb72 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/dc93cd10c6804dfab9054e7d5269cb72 2023-04-25 09:54:14,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/60099b717f8441408ca83ded291f80fd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/60099b717f8441408ca83ded291f80fd 2023-04-25 09:54:14,187 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/90bf26b4494e4f12af7c9c97c0e054d3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/90bf26b4494e4f12af7c9c97c0e054d3 2023-04-25 09:54:14,188 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2c7a296062e14fa28fd654d669f59c23 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2c7a296062e14fa28fd654d669f59c23 2023-04-25 09:54:14,189 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a5dbb79e619b40af85ff83583a1b3f1c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a5dbb79e619b40af85ff83583a1b3f1c 2023-04-25 09:54:14,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/acd83434476142ff8b39e7331102bf18 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/acd83434476142ff8b39e7331102bf18 2023-04-25 09:54:14,191 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a83cec13323c4bac95e5b62e197d2ac1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/a83cec13323c4bac95e5b62e197d2ac1 2023-04-25 09:54:14,192 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/393212c5c30948639e89a64973f2ebb4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/393212c5c30948639e89a64973f2ebb4 2023-04-25 09:54:14,193 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/ce096422ed404207a53decdd96a81fa2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/ce096422ed404207a53decdd96a81fa2 2023-04-25 09:54:14,194 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0c6cca26c904482c9e7b686b0fd0aa7a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/0c6cca26c904482c9e7b686b0fd0aa7a 2023-04-25 09:54:14,195 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/6a7aba7a4abc44b9a8b44e694fc1a1d9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/6a7aba7a4abc44b9a8b44e694fc1a1d9 2023-04-25 09:54:14,196 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/f6d356adf1ca4031896d86a6d236a614 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/f6d356adf1ca4031896d86a6d236a614 2023-04-25 09:54:14,197 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/46844d5cbe134134b483f13a44b7729e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/46844d5cbe134134b483f13a44b7729e 2023-04-25 09:54:14,199 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2d1a9ad9eed1495d8038dcc8d5ac6a8d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/2d1a9ad9eed1495d8038dcc8d5ac6a8d 2023-04-25 09:54:14,200 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5be81f79b27d45d3b0cf9ef492faa5ec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5be81f79b27d45d3b0cf9ef492faa5ec 2023-04-25 09:54:14,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9ceddbce7cf24871b638557585621e93 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/9ceddbce7cf24871b638557585621e93 2023-04-25 09:54:14,202 DEBUG [StoreCloser-TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e09f757de1d5475d91a8c62ec580fe16 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e09f757de1d5475d91a8c62ec580fe16 2023-04-25 09:54:14,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/recovered.edits/697.seqid, newMaxSeqId=697, maxSeqId=1 2023-04-25 09:54:14,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b. 2023-04-25 09:54:14,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6d622ed976f9d94b52b9c4f656801b6b: 2023-04-25 09:54:14,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:14,211 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6d622ed976f9d94b52b9c4f656801b6b, regionState=CLOSED 2023-04-25 09:54:14,211 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416454211"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416454211"}]},"ts":"1682416454211"} 2023-04-25 09:54:14,214 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-25 09:54:14,214 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 6d622ed976f9d94b52b9c4f656801b6b, server=jenkins-hbase4.apache.org,39407,1682416417123 in 1.6030 sec 2023-04-25 09:54:14,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-25 09:54:14,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6d622ed976f9d94b52b9c4f656801b6b, UNASSIGN in 1.6100 sec 2023-04-25 09:54:14,217 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416454216"}]},"ts":"1682416454216"} 2023-04-25 09:54:14,218 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 09:54:14,219 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 09:54:14,221 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8060 sec 2023-04-25 09:54:14,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-25 09:54:14,530 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-25 09:54:14,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 09:54:14,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,542 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,543 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 09:54:14,548 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:14,550 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/recovered.edits] 2023-04-25 09:54:14,553 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1a64be2302164706a5f850be93f16553 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/1a64be2302164706a5f850be93f16553 2023-04-25 09:54:14,555 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cdf52bfa4244485e90ee9b4ccfdc097d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/A/cdf52bfa4244485e90ee9b4ccfdc097d 2023-04-25 09:54:14,557 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1b645ef52b5a48a1af1b6507bb48266f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/1b645ef52b5a48a1af1b6507bb48266f 2023-04-25 09:54:14,558 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b4968ad5cbcd4340a7fa589642f77f93 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/B/b4968ad5cbcd4340a7fa589642f77f93 2023-04-25 09:54:14,561 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5afe68810c674ccc86b17f78e8b003d7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/5afe68810c674ccc86b17f78e8b003d7 2023-04-25 09:54:14,562 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e28ebd997a534e48a612c6fb98064b06 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/C/e28ebd997a534e48a612c6fb98064b06 2023-04-25 09:54:14,565 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/recovered.edits/697.seqid to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b/recovered.edits/697.seqid 2023-04-25 09:54:14,566 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6d622ed976f9d94b52b9c4f656801b6b 2023-04-25 09:54:14,566 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:54:14,568 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,589 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 09:54:14,630 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 09:54:14,631 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,631 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 09:54:14,632 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682416454631"}]},"ts":"9223372036854775807"} 2023-04-25 09:54:14,634 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 09:54:14,634 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6d622ed976f9d94b52b9c4f656801b6b, NAME => 'TestAcidGuarantees,,1682416420762.6d622ed976f9d94b52b9c4f656801b6b.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 09:54:14,634 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 09:54:14,634 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682416454634"}]},"ts":"9223372036854775807"} 2023-04-25 09:54:14,636 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 09:54:14,638 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 103 msec 2023-04-25 09:54:14,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-25 09:54:14,646 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-25 09:54:14,662 INFO [Listener at localhost/33705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=229 (was 204) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:39407-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5e92c536-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5e92c536-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1755476648_15 at /127.0.0.1:40906 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5e92c536-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5e92c536-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=539 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=392 (was 304) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4225 (was 4912) 2023-04-25 09:54:14,673 INFO [Listener at localhost/33705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=392, ProcessCount=170, AvailableMemoryMB=4225 2023-04-25 09:54:14,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:54:14,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:54:14,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:14,677 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:54:14,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-25 09:54:14,678 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:54:14,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 09:54:14,680 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:14,680 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 empty. 2023-04-25 09:54:14,681 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:14,681 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:54:14,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 09:54:14,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 09:54:15,090 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:54:15,092 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5687fcb9918272e953b19ef4123710b6, NAME => 'TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:54:15,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 09:54:15,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:54:15,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5687fcb9918272e953b19ef4123710b6, disabling compactions & flushes 2023-04-25 09:54:15,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. after waiting 0 ms 2023-04-25 09:54:15,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,499 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,499 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:15,501 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:54:15,502 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416455502"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416455502"}]},"ts":"1682416455502"} 2023-04-25 09:54:15,504 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:54:15,505 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:54:15,505 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416455505"}]},"ts":"1682416455505"} 2023-04-25 09:54:15,506 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 09:54:15,509 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, ASSIGN}] 2023-04-25 09:54:15,511 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, ASSIGN 2023-04-25 09:54:15,511 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:54:15,662 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:15,663 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416455662"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416455662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416455662"}]},"ts":"1682416455662"} 2023-04-25 09:54:15,664 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:54:15,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 09:54:15,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5687fcb9918272e953b19ef4123710b6, NAME => 'TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:54:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:54:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,822 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,823 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A 2023-04-25 09:54:15,824 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A 2023-04-25 09:54:15,824 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:15,824 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5687fcb9918272e953b19ef4123710b6 columnFamilyName A 2023-04-25 09:54:15,825 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(310): Store=5687fcb9918272e953b19ef4123710b6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:15,825 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,826 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B 2023-04-25 09:54:15,826 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B 2023-04-25 09:54:15,826 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:15,827 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5687fcb9918272e953b19ef4123710b6 columnFamilyName B 2023-04-25 09:54:15,827 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(310): Store=5687fcb9918272e953b19ef4123710b6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:15,827 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,829 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C 2023-04-25 09:54:15,829 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C 2023-04-25 09:54:15,829 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:15,829 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5687fcb9918272e953b19ef4123710b6 columnFamilyName C 2023-04-25 09:54:15,829 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(310): Store=5687fcb9918272e953b19ef4123710b6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:15,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:54:15,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:15,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:54:15,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5687fcb9918272e953b19ef4123710b6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63753975, jitterRate=-0.04999174177646637}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:54:15,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:15,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., pid=18, masterSystemTime=1682416455816 2023-04-25 09:54:15,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:15,840 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:15,840 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416455840"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416455840"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416455840"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416455840"}]},"ts":"1682416455840"} 2023-04-25 09:54:15,843 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-25 09:54:15,843 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 in 177 msec 2023-04-25 09:54:15,845 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-25 09:54:15,845 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, ASSIGN in 334 msec 2023-04-25 09:54:15,846 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:54:15,846 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416455846"}]},"ts":"1682416455846"} 2023-04-25 09:54:15,847 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 09:54:15,849 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:54:15,850 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-04-25 09:54:16,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-25 09:54:16,783 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-25 09:54:16,784 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x5b8d0eb2 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:16,789 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52274650, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:16,790 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:16,795 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:16,796 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:54:16,798 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:54:16,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:54:16,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:54:16,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:16,829 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 09:54:16,830 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:54:16,830 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 09:54:16,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 09:54:16,850 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, REOPEN/MOVE}] 2023-04-25 09:54:16,851 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, REOPEN/MOVE 2023-04-25 09:54:16,852 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:16,852 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416456852"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416456852"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416456852"}]},"ts":"1682416456852"} 2023-04-25 09:54:16,854 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:54:17,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5687fcb9918272e953b19ef4123710b6, disabling compactions & flushes 2023-04-25 09:54:17,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. after waiting 0 ms 2023-04-25 09:54:17,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,011 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 09:54:17,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:17,012 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5687fcb9918272e953b19ef4123710b6 to self. 2023-04-25 09:54:17,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,014 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=CLOSED 2023-04-25 09:54:17,014 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416457014"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416457014"}]},"ts":"1682416457014"} 2023-04-25 09:54:17,017 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-25 09:54:17,017 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 in 161 msec 2023-04-25 09:54:17,017 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=true 2023-04-25 09:54:17,168 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,168 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416457168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416457168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416457168"}]},"ts":"1682416457168"} 2023-04-25 09:54:17,170 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:54:17,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5687fcb9918272e953b19ef4123710b6, NAME => 'TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:54:17,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:54:17,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,329 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,348 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A 2023-04-25 09:54:17,348 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A 2023-04-25 09:54:17,348 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:17,351 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5687fcb9918272e953b19ef4123710b6 columnFamilyName A 2023-04-25 09:54:17,353 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(310): Store=5687fcb9918272e953b19ef4123710b6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:17,353 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,354 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B 2023-04-25 09:54:17,354 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B 2023-04-25 09:54:17,355 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:17,355 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5687fcb9918272e953b19ef4123710b6 columnFamilyName B 2023-04-25 09:54:17,355 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(310): Store=5687fcb9918272e953b19ef4123710b6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:17,356 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,356 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C 2023-04-25 09:54:17,356 DEBUG [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C 2023-04-25 09:54:17,357 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:17,357 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5687fcb9918272e953b19ef4123710b6 columnFamilyName C 2023-04-25 09:54:17,358 INFO [StoreOpener-5687fcb9918272e953b19ef4123710b6-1] regionserver.HStore(310): Store=5687fcb9918272e953b19ef4123710b6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:17,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:54:17,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5687fcb9918272e953b19ef4123710b6; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65795708, jitterRate=-0.019567549228668213}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:54:17,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:17,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., pid=23, masterSystemTime=1682416457322 2023-04-25 09:54:17,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,366 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,366 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416457365"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416457365"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416457365"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416457365"}]},"ts":"1682416457365"} 2023-04-25 09:54:17,369 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-25 09:54:17,369 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 in 197 msec 2023-04-25 09:54:17,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-25 09:54:17,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, REOPEN/MOVE in 519 msec 2023-04-25 09:54:17,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-25 09:54:17,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 538 msec 2023-04-25 09:54:17,374 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 563 msec 2023-04-25 09:54:17,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-25 09:54:17,384 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x051c62b6 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,388 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21281a37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,390 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,393 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,394 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,400 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,401 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,406 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,406 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,409 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,409 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,413 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,414 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,418 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,419 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,422 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,423 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x58e27a32 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:17,427 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a7b66a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:17,436 DEBUG [hconnection-0x45329129-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,436 DEBUG [hconnection-0x28a360da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,438 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:17,440 DEBUG [hconnection-0x27946df8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,447 DEBUG [hconnection-0x1a1e03e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,448 DEBUG [hconnection-0x45635d31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,448 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,449 DEBUG [hconnection-0x176c4c2d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,449 DEBUG [hconnection-0x3e6d736b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,452 DEBUG [hconnection-0x2d2c0dcd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,452 DEBUG [hconnection-0x3b0bae0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:17,454 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,454 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,454 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,456 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,461 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,461 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,462 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:17,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:17,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:17,473 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:17,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:17,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:17,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:17,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:17,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:17,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:17,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:17,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:17,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:17,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:17,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:17,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:17,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:17,489 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,489 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:17,489 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:17,489 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:17,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:17,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:17,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:17,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762e9d25[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:17,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:17,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,499 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:17,499 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:17,499 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:17,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:17,500 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:17,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,501 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:17,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:17,504 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:17,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:17,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:17,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:17,504 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:17,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,507 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,508 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,519 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416517514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:17,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:17,520 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,520 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:17,520 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:17,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682416457507, max timestamp=1682416457518Num uniques 859; 2023-04-25 09:54:17,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682416457507, max timestamp=1682416457518Num uniques 859; 2023-04-25 09:54:17,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:17,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:17,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416457507, max timestamp=1682416457518Num uniques 859; 2023-04-25 09:54:17,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:54:17,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:17,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416517553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416517569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416517569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416517569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,620 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:17,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:17,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:17,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:17,627 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:17,628 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,628 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:17,628 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:17,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416517624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,628 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,628 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,633 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,633 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:17,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:17,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:17,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,635 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:17,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:17,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:17,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:17,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,638 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:17,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55684823[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:17,640 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:17,640 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,642 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:17,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:17,645 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:17,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:17,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:17,655 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:17,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:17,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:17,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:17,655 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:17,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,658 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,659 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:17,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:17,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,662 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:17,662 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:17,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416517677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416517677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416517677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416517678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,763 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:17,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:17,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:17,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:17,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:17,772 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:17,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:17,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:17,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,774 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,774 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,775 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,775 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:17,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:17,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:17,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:17,780 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:17,780 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,780 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:17,788 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,788 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:17,788 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:17,788 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:17,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:17,791 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,792 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,792 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:17,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1b2a7f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:17,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:17,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,795 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,796 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:17,796 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:17,796 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:17,796 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,796 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:17,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:17,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:17,801 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:17,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:17,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:17,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:17,801 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:17,803 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,803 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,803 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:17,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,810 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:17,810 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:17,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416517830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416517880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416517881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416517881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:17,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416517882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,910 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:17,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:17,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:17,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:17,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:17,923 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:17,923 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,924 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:17,924 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:17,924 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,924 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,925 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:17,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:17,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:17,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:17,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:17,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:17,945 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252f316d92871d4ae494fe3b1e87f2d653 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252f316d92871d4ae494fe3b1e87f2d653 2023-04-25 09:54:17,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:17,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:17,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:17,972 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,972 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:17,972 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:17,972 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:17,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:17,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:17,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:17,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:17,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4b8708[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:17,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:17,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:17,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:17,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:17,983 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/f3b49d54998442109692a176a7ca91b8 2023-04-25 09:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:17,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:17,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:17,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:17,992 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:17,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:17,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:17,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:17,992 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:17,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:17,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:17,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:17,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:17,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:17,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:17,998 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,000 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:18,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:18,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,001 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:18,001 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:18,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/dcff060b05744e159e230896ff283124 2023-04-25 09:54:18,101 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:18,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:18,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:18,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:18,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:18,113 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:18,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:18,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:18,120 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,120 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,121 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:18,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:18,133 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:18,133 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,133 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:18,134 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,134 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:18,134 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:18,135 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:18,135 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:18,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416518138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,139 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:18,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@780efad1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:18,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:18,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,143 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:18,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:18,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:18,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:18,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:18,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,144 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:18,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:18,148 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:18,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:18,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:18,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:18,148 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:18,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,154 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:18,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,160 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,160 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416518185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416518186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416518186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416518192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,260 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:18,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:18,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:18,265 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,266 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,266 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:18,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:18,268 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,268 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:18,268 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:18,268 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:18,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:18,270 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:18,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f309ca3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:18,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:18,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,274 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:18,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:18,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:18,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:18,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:18,278 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:18,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:18,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:18,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:18,278 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:18,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:18,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:18,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:18,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,285 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:18,285 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:18,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,385 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:18,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:18,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:18,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:18,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:18,389 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:18,389 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,389 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:18,389 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:18,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,393 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:18,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:18,394 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:18,394 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:18,395 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,395 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:18,395 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:18,395 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:18,397 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,397 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,397 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,398 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:18,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cdb1be8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:18,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:18,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,400 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:18,400 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:18,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:18,401 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:18,404 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:18,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:18,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:18,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:18,404 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:18,406 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,406 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,406 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:18,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,407 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:18,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:18,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:18,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,412 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:18,412 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:18,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:18,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c90619f3507745d9af68ce1e6a8d4844 2023-04-25 09:54:18,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/f3b49d54998442109692a176a7ca91b8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f3b49d54998442109692a176a7ca91b8 2023-04-25 09:54:18,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f3b49d54998442109692a176a7ca91b8, entries=150, sequenceid=16, filesize=25.1 K 2023-04-25 09:54:18,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/dcff060b05744e159e230896ff283124 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/dcff060b05744e159e230896ff283124 2023-04-25 09:54:18,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/dcff060b05744e159e230896ff283124, entries=150, sequenceid=16, filesize=11.6 K 2023-04-25 09:54:18,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c90619f3507745d9af68ce1e6a8d4844 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c90619f3507745d9af68ce1e6a8d4844 2023-04-25 09:54:18,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c90619f3507745d9af68ce1e6a8d4844, entries=150, sequenceid=16, filesize=11.6 K 2023-04-25 09:54:18,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=131.55 KB/134706 for 5687fcb9918272e953b19ef4123710b6 in 987ms, sequenceid=16, compaction requested=false 2023-04-25 09:54:18,470 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 09:54:18,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:18,512 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:18,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:18,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:18,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:18,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:18,517 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:18,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:18,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:18,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,519 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:18,519 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:18,519 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,519 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:18,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:18,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:18,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:18,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:18,521 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:18,521 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:18,521 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:18,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:18,524 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:18,524 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:18,525 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-25 09:54:18,525 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:18,525 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:18,525 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:18,525 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:18,525 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:18,525 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:18,566 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425004ce414fbb74414ba56c218a746d0e2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425004ce414fbb74414ba56c218a746d0e2 2023-04-25 09:54:18,577 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/3a2b24ed1db74685a8b1fd54bd3a8a11 2023-04-25 09:54:18,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:18,642 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:18,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416518661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416518688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416518688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416518689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416518700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416518764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416518968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:18,998 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/5c39df869d1045309736e97c5dc3ccc4 2023-04-25 09:54:19,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416519271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,438 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.10 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/b9c3be2174ac4eb0a1d4a12dba150fa0 2023-04-25 09:54:19,445 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/3a2b24ed1db74685a8b1fd54bd3a8a11 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/3a2b24ed1db74685a8b1fd54bd3a8a11 2023-04-25 09:54:19,450 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/3a2b24ed1db74685a8b1fd54bd3a8a11, entries=150, sequenceid=52, filesize=25.1 K 2023-04-25 09:54:19,451 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/5c39df869d1045309736e97c5dc3ccc4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/5c39df869d1045309736e97c5dc3ccc4 2023-04-25 09:54:19,458 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/5c39df869d1045309736e97c5dc3ccc4, entries=150, sequenceid=52, filesize=11.6 K 2023-04-25 09:54:19,459 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/b9c3be2174ac4eb0a1d4a12dba150fa0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b9c3be2174ac4eb0a1d4a12dba150fa0 2023-04-25 09:54:19,464 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b9c3be2174ac4eb0a1d4a12dba150fa0, entries=150, sequenceid=52, filesize=11.6 K 2023-04-25 09:54:19,465 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134706, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for 5687fcb9918272e953b19ef4123710b6 in 940ms, sequenceid=52, compaction requested=false 2023-04-25 09:54:19,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:19,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:19,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:19,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:19,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:19,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:19,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8ea6bc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:19,470 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:19,470 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:19,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:19,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:19,477 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:19,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:19,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:19,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:19,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:19,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:19,480 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:19,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:19,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:19,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:19,480 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:19,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:19,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:19,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:19,483 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,484 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,487 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:19,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:19,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:19,487 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,488 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:19,488 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:19,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:19,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:19,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:19,588 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:19,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:19,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:19,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:19,592 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:19,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:19,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:19,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:19,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:19,595 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:19,595 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:19,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:19,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:19,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:19,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:19,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:19,598 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:19,598 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:19,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:19,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:19,598 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:19,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:19,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:19,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:19,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:19,599 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:19,599 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:19,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:19,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:19,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416459706, max timestamp=1682416459710Num uniques 859; 2023-04-25 09:54:19,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416459706, max timestamp=1682416459710Num uniques 859; 2023-04-25 09:54:19,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:19,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:19,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416459706, max timestamp=1682416459710Num uniques 859; 2023-04-25 09:54:19,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416519726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416519726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416519727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416519729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,746 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 09:54:19,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416519774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416519830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416519831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416519831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416519831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,018 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255d8839bc0dc84057b107c48b6c805c9c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255d8839bc0dc84057b107c48b6c805c9c 2023-04-25 09:54:20,023 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/9c74fb5a3af843d89334cec6246139a2 2023-04-25 09:54:20,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416520035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416520035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416520036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416520036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,059 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/17ebfe6943ae458bbddea87750f39dfc 2023-04-25 09:54:20,073 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/7cd9008a628540d980753e22ead06b0c 2023-04-25 09:54:20,079 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/9c74fb5a3af843d89334cec6246139a2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9c74fb5a3af843d89334cec6246139a2 2023-04-25 09:54:20,083 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9c74fb5a3af843d89334cec6246139a2, entries=150, sequenceid=62, filesize=25.1 K 2023-04-25 09:54:20,084 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/17ebfe6943ae458bbddea87750f39dfc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/17ebfe6943ae458bbddea87750f39dfc 2023-04-25 09:54:20,090 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/17ebfe6943ae458bbddea87750f39dfc, entries=150, sequenceid=62, filesize=11.6 K 2023-04-25 09:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/7cd9008a628540d980753e22ead06b0c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7cd9008a628540d980753e22ead06b0c 2023-04-25 09:54:20,100 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7cd9008a628540d980753e22ead06b0c, entries=150, sequenceid=62, filesize=11.6 K 2023-04-25 09:54:20,102 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5687fcb9918272e953b19ef4123710b6 in 504ms, sequenceid=62, compaction requested=true 2023-04-25 09:54:20,102 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:20,102 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:20,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:20,104 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d9014e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:20,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:20,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:20,108 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,108 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:20,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:20,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:20,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:20,113 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:20,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:20,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:20,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:20,113 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:20,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:20,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:20,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:20,116 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:20,119 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:20,121 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:20,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:20,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:20,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:20,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:20,122 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:20,122 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:20,222 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:20,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:20,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:20,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:20,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:20,226 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:20,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:20,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:20,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:20,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:20,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:20,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:20,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:20,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:20,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:20,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:20,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:20,230 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:20,230 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:20,230 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-25 09:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:20,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:20,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416520368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416520369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416520370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416520371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416520473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416520476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416520476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416520477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,645 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425235d5fa099774292bfaf148ebe4ef0b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425235d5fa099774292bfaf148ebe4ef0b1 2023-04-25 09:54:20,655 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/c02aca29e13e4d73b67b04a20332a197 2023-04-25 09:54:20,666 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/b22bcff7f1924195b6bce80aa4d4ce79 2023-04-25 09:54:20,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416520678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416520682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416520682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416520682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416520790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416520980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416520988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416520993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:20,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:20,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416520994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,083 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/54c499fecd2f44edb5a438f2d491dbff 2023-04-25 09:54:21,088 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/c02aca29e13e4d73b67b04a20332a197 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/c02aca29e13e4d73b67b04a20332a197 2023-04-25 09:54:21,093 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/c02aca29e13e4d73b67b04a20332a197, entries=150, sequenceid=97, filesize=25.1 K 2023-04-25 09:54:21,094 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/b22bcff7f1924195b6bce80aa4d4ce79 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b22bcff7f1924195b6bce80aa4d4ce79 2023-04-25 09:54:21,100 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b22bcff7f1924195b6bce80aa4d4ce79, entries=150, sequenceid=97, filesize=11.6 K 2023-04-25 09:54:21,101 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/54c499fecd2f44edb5a438f2d491dbff as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/54c499fecd2f44edb5a438f2d491dbff 2023-04-25 09:54:21,109 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/54c499fecd2f44edb5a438f2d491dbff, entries=150, sequenceid=97, filesize=11.6 K 2023-04-25 09:54:21,110 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 5687fcb9918272e953b19ef4123710b6 in 880ms, sequenceid=97, compaction requested=true 2023-04-25 09:54:21,110 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:21,110 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:21,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:21,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:21,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:21,112 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e674cd2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:21,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:21,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:21,117 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,117 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:21,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:21,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:21,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:21,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:21,125 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:21,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:21,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:21,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:21,125 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:21,128 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,128 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,128 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:21,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,131 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:21,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:21,134 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:21,134 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:21,134 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:21,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:21,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:21,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:21,234 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:21,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:21,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:21,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:21,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:21,238 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:21,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:21,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:21,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:21,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,241 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:21,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:21,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,242 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,242 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:21,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:21,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:21,243 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:21,244 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:21,244 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:21,244 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:21,244 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:21,244 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:21,263 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e2dd56ce1c5f4df0b10279861a382d6a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e2dd56ce1c5f4df0b10279861a382d6a 2023-04-25 09:54:21,274 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/480f3b56f2b94a27b0493d4fa264bc2c 2023-04-25 09:54:21,289 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/837985e72e1c486f9175721fe63634a1 2023-04-25 09:54:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:21,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682416461497, max timestamp=1682416461513Num uniques 859; 2023-04-25 09:54:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682416461497, max timestamp=1682416461513Num uniques 859; 2023-04-25 09:54:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 09:54:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682416461497, max timestamp=1682416461513Num uniques 859; 2023-04-25 09:54:21,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 09:54:21,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 09:54:21,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416521527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416521527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416521528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416521535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416521636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416521637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416521637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416521638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,701 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c4810d9a561549708d3b3cb2ff79a4e1 2023-04-25 09:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/480f3b56f2b94a27b0493d4fa264bc2c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/480f3b56f2b94a27b0493d4fa264bc2c 2023-04-25 09:54:21,710 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/480f3b56f2b94a27b0493d4fa264bc2c, entries=150, sequenceid=107, filesize=25.1 K 2023-04-25 09:54:21,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/837985e72e1c486f9175721fe63634a1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/837985e72e1c486f9175721fe63634a1 2023-04-25 09:54:21,719 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/837985e72e1c486f9175721fe63634a1, entries=150, sequenceid=107, filesize=11.6 K 2023-04-25 09:54:21,721 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c4810d9a561549708d3b3cb2ff79a4e1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c4810d9a561549708d3b3cb2ff79a4e1 2023-04-25 09:54:21,725 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c4810d9a561549708d3b3cb2ff79a4e1, entries=150, sequenceid=107, filesize=11.6 K 2023-04-25 09:54:21,726 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5687fcb9918272e953b19ef4123710b6 in 483ms, sequenceid=107, compaction requested=true 2023-04-25 09:54:21,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:21,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:21,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:21,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:21,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:21,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af702ad[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:21,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:21,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,731 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:21,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:21,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:21,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:21,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,734 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,734 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:21,734 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:21,737 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:21,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:21,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:21,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:21,737 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:21,742 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,742 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,742 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,743 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,744 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:21,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:21,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:21,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:21,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:21,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:21,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,749 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:21,749 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:21,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:21,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:21,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:21,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=147.60 KB heapSize=401.39 KB 2023-04-25 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:21,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:21,850 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:21,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:21,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416521850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416521851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416521851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:21,856 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:21,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:21,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:21,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:21,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,858 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:21,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:21,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:21,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:21,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416521852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:21,863 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:21,863 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:21,863 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:21,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:21,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:21,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:21,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:21,870 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,871 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,871 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,873 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304253cde8ea2f9bc4e9da61c85517c1ad7f1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253cde8ea2f9bc4e9da61c85517c1ad7f1 2023-04-25 09:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:21,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@553fc09[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:21,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:21,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:21,877 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:21,877 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:21,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:21,878 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:21,884 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:21,884 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:21,884 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:21,884 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:21,884 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:21,885 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/4ae2940d9b2748ca909628bffcdd0410 2023-04-25 09:54:21,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:21,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:21,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,887 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:21,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:21,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:21,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:21,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:21,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:21,891 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:21,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:21,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:21,891 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:21,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/6fb9c16345f5471dabe65bc48d0e9dfb 2023-04-25 09:54:21,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/ca28cce4e0da4a8fa85d49f71392aff6 2023-04-25 09:54:21,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/4ae2940d9b2748ca909628bffcdd0410 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/4ae2940d9b2748ca909628bffcdd0410 2023-04-25 09:54:21,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/4ae2940d9b2748ca909628bffcdd0410, entries=250, sequenceid=142, filesize=38.9 K 2023-04-25 09:54:21,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/6fb9c16345f5471dabe65bc48d0e9dfb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6fb9c16345f5471dabe65bc48d0e9dfb 2023-04-25 09:54:21,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6fb9c16345f5471dabe65bc48d0e9dfb, entries=150, sequenceid=142, filesize=11.8 K 2023-04-25 09:54:21,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/ca28cce4e0da4a8fa85d49f71392aff6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/ca28cce4e0da4a8fa85d49f71392aff6 2023-04-25 09:54:21,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/ca28cce4e0da4a8fa85d49f71392aff6, entries=150, sequenceid=142, filesize=11.8 K 2023-04-25 09:54:21,933 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~421.27 KB/431376, currentSize=33.54 KB/34350 for 5687fcb9918272e953b19ef4123710b6 in 92ms, sequenceid=142, compaction requested=true 2023-04-25 09:54:21,933 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:21,933 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:54:21,933 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:21,933 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:21,933 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:54:21,934 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:54:21,937 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 168610 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:54:21,937 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:21,937 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,938 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f3b49d54998442109692a176a7ca91b8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/3a2b24ed1db74685a8b1fd54bd3a8a11, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9c74fb5a3af843d89334cec6246139a2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/c02aca29e13e4d73b67b04a20332a197, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/480f3b56f2b94a27b0493d4fa264bc2c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/4ae2940d9b2748ca909628bffcdd0410] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=164.7 K 2023-04-25 09:54:21,938 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:54:21,938 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:21,938 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f3b49d54998442109692a176a7ca91b8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682416457464 2023-04-25 09:54:21,938 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:21,938 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/dcff060b05744e159e230896ff283124, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/5c39df869d1045309736e97c5dc3ccc4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/17ebfe6943ae458bbddea87750f39dfc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b22bcff7f1924195b6bce80aa4d4ce79, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/837985e72e1c486f9175721fe63634a1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6fb9c16345f5471dabe65bc48d0e9dfb] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=69.8 K 2023-04-25 09:54:21,938 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3a2b24ed1db74685a8b1fd54bd3a8a11, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682416457537 2023-04-25 09:54:21,939 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting dcff060b05744e159e230896ff283124, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682416457464 2023-04-25 09:54:21,939 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9c74fb5a3af843d89334cec6246139a2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682416458654 2023-04-25 09:54:21,939 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5c39df869d1045309736e97c5dc3ccc4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682416457537 2023-04-25 09:54:21,939 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c02aca29e13e4d73b67b04a20332a197, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682416459724 2023-04-25 09:54:21,939 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 17ebfe6943ae458bbddea87750f39dfc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682416458654 2023-04-25 09:54:21,940 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 480f3b56f2b94a27b0493d4fa264bc2c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682416460342 2023-04-25 09:54:21,940 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4ae2940d9b2748ca909628bffcdd0410, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682416461526 2023-04-25 09:54:21,940 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b22bcff7f1924195b6bce80aa4d4ce79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682416459724 2023-04-25 09:54:21,941 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 837985e72e1c486f9175721fe63634a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682416460342 2023-04-25 09:54:21,941 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6fb9c16345f5471dabe65bc48d0e9dfb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682416461526 2023-04-25 09:54:21,960 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#142 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:21,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:21,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:21,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:21,964 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#141 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:21,992 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:21,994 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/e9611b555890443abeeb4d36b580fcab as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/e9611b555890443abeeb4d36b580fcab 2023-04-25 09:54:21,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:22,002 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425662a93f9003b485db896b9f5f293c3b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425662a93f9003b485db896b9f5f293c3b6 2023-04-25 09:54:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:22,005 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:22,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:22,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:22,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,007 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into e9611b555890443abeeb4d36b580fcab(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:22,007 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:22,007 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=10, startTime=1682416461933; duration=0sec 2023-04-25 09:54:22,007 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:22,007 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:54:22,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416461984, max timestamp=1682416462012Num uniques 859; 2023-04-25 09:54:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416461984, max timestamp=1682416462012Num uniques 859; 2023-04-25 09:54:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:54:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416461984, max timestamp=1682416462012Num uniques 859; 2023-04-25 09:54:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:54:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:54:22,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:22,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:22,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:22,018 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:54:22,018 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:22,019 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,019 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c90619f3507745d9af68ce1e6a8d4844, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b9c3be2174ac4eb0a1d4a12dba150fa0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7cd9008a628540d980753e22ead06b0c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/54c499fecd2f44edb5a438f2d491dbff, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c4810d9a561549708d3b3cb2ff79a4e1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/ca28cce4e0da4a8fa85d49f71392aff6] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=69.8 K 2023-04-25 09:54:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:22,019 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,019 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:22,019 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:22,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,021 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c90619f3507745d9af68ce1e6a8d4844, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682416457464 2023-04-25 09:54:22,021 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b9c3be2174ac4eb0a1d4a12dba150fa0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1682416457537 2023-04-25 09:54:22,021 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7cd9008a628540d980753e22ead06b0c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682416458654 2023-04-25 09:54:22,022 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 54c499fecd2f44edb5a438f2d491dbff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682416459724 2023-04-25 09:54:22,022 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c4810d9a561549708d3b3cb2ff79a4e1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1682416460342 2023-04-25 09:54:22,023 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ca28cce4e0da4a8fa85d49f71392aff6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682416461526 2023-04-25 09:54:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,033 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,038 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/6b0fdfe64a86432290cdc115d3ad0aea 2023-04-25 09:54:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:22,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ad5e9c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:22,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:22,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,040 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#144 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:22,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:22,042 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,042 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:22,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:22,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:22,048 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:22,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:22,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:22,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:22,048 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:22,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416522040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,051 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,051 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,051 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,053 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416522047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416522050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416522050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:22,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,057 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:22,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,057 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:22,058 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,069 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/683ff37db2874d119d4335d18e0e393b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/683ff37db2874d119d4335d18e0e393b 2023-04-25 09:54:22,074 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into 683ff37db2874d119d4335d18e0e393b(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:22,074 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:22,074 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=10, startTime=1682416461933; duration=0sec 2023-04-25 09:54:22,075 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:22,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416522150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,158 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:22,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:22,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:22,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416522156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416522156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416522157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:22,163 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:22,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:22,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:22,164 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,164 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:22,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:22,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:22,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:22,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:22,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:22,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:22,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:22,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c188687[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:22,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:22,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:22,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:22,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:22,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:22,178 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:22,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:22,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:22,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:22,178 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:22,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,182 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:22,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,189 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:22,190 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:22,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,290 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:22,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:22,293 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:22,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:22,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:22,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:22,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:22,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:22,297 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,297 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:22,297 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:22,297 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:22,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e581dc3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:22,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:22,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:22,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:22,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:22,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:22,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:22,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:22,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:22,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:22,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:22,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:22,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,312 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:22,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:22,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,312 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:22,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416522352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416522362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416522363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416522364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,412 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:22,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:22,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:22,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:22,416 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:22,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:22,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:22,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:22,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:22,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:22,419 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,419 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:22,419 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:22,419 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:22,426 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24711db5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:22,428 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:22,428 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:22,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,431 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:22,433 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:22,433 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:22,433 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:22,433 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:22,433 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:22,435 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,435 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,435 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:22,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:22,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:22,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:22,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:22,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:22,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:22,441 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:22,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,441 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:22,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:22,472 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/8451102c859f433991c3ec4f347ed22a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8451102c859f433991c3ec4f347ed22a 2023-04-25 09:54:22,477 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 8451102c859f433991c3ec4f347ed22a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:22,477 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:22,477 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=10, startTime=1682416461933; duration=0sec 2023-04-25 09:54:22,478 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:22,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/1833e4c727dd432f9004c51940e8bcc8 2023-04-25 09:54:22,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/8c8fa2c315f640fa98ca67e6fc80da0a 2023-04-25 09:54:22,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/6b0fdfe64a86432290cdc115d3ad0aea as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/6b0fdfe64a86432290cdc115d3ad0aea 2023-04-25 09:54:22,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/6b0fdfe64a86432290cdc115d3ad0aea, entries=300, sequenceid=153, filesize=45.8 K 2023-04-25 09:54:22,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/1833e4c727dd432f9004c51940e8bcc8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1833e4c727dd432f9004c51940e8bcc8 2023-04-25 09:54:22,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1833e4c727dd432f9004c51940e8bcc8, entries=150, sequenceid=153, filesize=11.8 K 2023-04-25 09:54:22,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/8c8fa2c315f640fa98ca67e6fc80da0a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8c8fa2c315f640fa98ca67e6fc80da0a 2023-04-25 09:54:22,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8c8fa2c315f640fa98ca67e6fc80da0a, entries=150, sequenceid=153, filesize=11.8 K 2023-04-25 09:54:22,522 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 5687fcb9918272e953b19ef4123710b6 in 560ms, sequenceid=153, compaction requested=false 2023-04-25 09:54:22,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:22,541 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:22,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:22,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:22,545 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:22,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:22,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:22,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:22,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:22,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:22,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:22,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:22,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:22,549 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:22,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:22,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:22,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416522668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416522669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416522669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416522671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416522772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416522773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416522775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44144 deadline: 1682416522802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,804 DEBUG [Thread-918] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:22,963 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257fb75cfa99ca4f3698275f99fbf6ccfd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257fb75cfa99ca4f3698275f99fbf6ccfd 2023-04-25 09:54:22,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416522974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416522975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416522977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416523175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416523279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416523281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416523283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,369 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/505eba8f6ae143658468a02daee4c4af 2023-04-25 09:54:23,781 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/2035c8ecf3be4f6d8e50bc78da22ad4b 2023-04-25 09:54:23,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416523784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416523787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:23,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:23,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416523790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416524184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,201 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/6c50f56c426d45329c54be0d405a9660 2023-04-25 09:54:24,206 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/505eba8f6ae143658468a02daee4c4af as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/505eba8f6ae143658468a02daee4c4af 2023-04-25 09:54:24,210 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/505eba8f6ae143658468a02daee4c4af, entries=150, sequenceid=190, filesize=25.3 K 2023-04-25 09:54:24,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/2035c8ecf3be4f6d8e50bc78da22ad4b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2035c8ecf3be4f6d8e50bc78da22ad4b 2023-04-25 09:54:24,215 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2035c8ecf3be4f6d8e50bc78da22ad4b, entries=150, sequenceid=190, filesize=11.8 K 2023-04-25 09:54:24,216 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/6c50f56c426d45329c54be0d405a9660 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/6c50f56c426d45329c54be0d405a9660 2023-04-25 09:54:24,220 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/6c50f56c426d45329c54be0d405a9660, entries=150, sequenceid=190, filesize=11.8 K 2023-04-25 09:54:24,220 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for 5687fcb9918272e953b19ef4123710b6 in 1671ms, sequenceid=190, compaction requested=true 2023-04-25 09:54:24,220 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:24,221 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:24,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:24,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:24,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@666aee19[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:24,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:24,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,225 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:24,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:24,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:24,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:24,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:24,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:24,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:24,231 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:24,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:24,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:24,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:24,231 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:24,232 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,234 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,240 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:24,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:24,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:24,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:24,241 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:24,241 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:24,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:24,241 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:24,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:24,341 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:24,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:24,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:24,344 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:24,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:24,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:24,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:24,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,347 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:24,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:24,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:24,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:24,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:24,349 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:24,349 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:24,349 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:24,349 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:24,350 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:24,350 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:24,350 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:24,350 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:24,350 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:24,389 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304250a49181156124e10b87cac263d24d204 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250a49181156124e10b87cac263d24d204 2023-04-25 09:54:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:24,789 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:24,797 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=200, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/5352c271c2af4be0b751bda610cc661f 2023-04-25 09:54:24,808 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/7d33c142fab84810a9d2e401bc10626e 2023-04-25 09:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416464810, max timestamp=1682416464822Num uniques 859; 2023-04-25 09:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416464810, max timestamp=1682416464822Num uniques 859; 2023-04-25 09:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:24,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:24,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416464817, max timestamp=1682416464822Num uniques 859; 2023-04-25 09:54:24,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:24,835 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c101d5933bd74117b764384554201e94 2023-04-25 09:54:24,840 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/5352c271c2af4be0b751bda610cc661f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/5352c271c2af4be0b751bda610cc661f 2023-04-25 09:54:24,844 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/5352c271c2af4be0b751bda610cc661f, entries=150, sequenceid=200, filesize=25.3 K 2023-04-25 09:54:24,845 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/7d33c142fab84810a9d2e401bc10626e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/7d33c142fab84810a9d2e401bc10626e 2023-04-25 09:54:24,850 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/7d33c142fab84810a9d2e401bc10626e, entries=150, sequenceid=200, filesize=11.8 K 2023-04-25 09:54:24,851 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c101d5933bd74117b764384554201e94 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c101d5933bd74117b764384554201e94 2023-04-25 09:54:24,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416524849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416524849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416524850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,855 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c101d5933bd74117b764384554201e94, entries=150, sequenceid=200, filesize=11.8 K 2023-04-25 09:54:24,855 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 5687fcb9918272e953b19ef4123710b6 in 506ms, sequenceid=200, compaction requested=true 2023-04-25 09:54:24,855 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:24,855 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:24,857 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e1e60ed[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:24,859 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:24,859 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,860 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:24,861 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:24,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:24,862 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:24,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:24,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:24,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:24,862 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:24,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:24,865 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,866 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:24,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:24,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:24,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:24,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:24,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:24,869 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:24,869 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:24,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:24,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=143.13 KB heapSize=405.30 KB 2023-04-25 09:54:24,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:24,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:54:24,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:24,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:54:24,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:24,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:54:24,969 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:24,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:24,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416524966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416524966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:24,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416524970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:24,973 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:24,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:24,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:24,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:24,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,975 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,975 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:24,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:24,975 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,975 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:24,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:24,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:24,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:24,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:24,977 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:24,977 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:24,978 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4056c4e8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:24,983 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:24,983 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:24,984 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:24,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:24,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:24,987 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:24,988 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:24,988 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:24,988 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:24,988 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:24,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,990 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:24,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:24,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:24,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:24,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:24,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:24,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:24,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:24,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:24,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:24,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:24,994 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:24,994 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:24,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416525071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416525071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416525072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,094 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,098 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,108 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,108 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,108 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,108 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce6e5e2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,116 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,116 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,117 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,117 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,117 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,120 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,121 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:25,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,124 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,124 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,224 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,227 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,229 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,229 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,229 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,236 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d4dd441[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,243 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,243 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,244 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,244 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,244 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,250 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,250 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,250 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:25,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,250 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,250 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,251 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,251 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,251 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416525273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416525274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416525274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,351 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,354 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,356 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,356 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,356 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,356 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,358 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,358 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,358 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,358 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,360 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1019de0d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,364 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,364 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,364 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,366 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,372 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,373 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,375 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,376 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,376 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,377 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,379 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c7c9ea296f1a42559841cc3985bca888 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c7c9ea296f1a42559841cc3985bca888 2023-04-25 09:54:25,379 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,379 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,379 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,379 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,380 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,380 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,380 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,380 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,380 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,380 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,380 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:25,480 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,485 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,487 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,488 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,488 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,489 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,490 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669e3962[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,497 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,497 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,497 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,497 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,498 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,498 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,498 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,501 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,501 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,505 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,510 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,510 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,510 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,510 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,510 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,511 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,510 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,511 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,511 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,511 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,511 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,511 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416525579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416525580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:25,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416525580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,611 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,615 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,616 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,617 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,617 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,619 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,619 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,619 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,619 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,621 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,621 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d0c09a6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,625 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,625 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,626 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,630 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,635 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,635 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,636 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,636 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,636 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,637 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,638 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,641 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:25,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,642 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,642 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,642 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,642 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,642 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,742 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,745 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,746 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,746 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,747 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,749 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,749 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,749 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,749 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e503d9a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,755 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,756 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,756 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,757 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,759 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,759 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,759 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,759 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,759 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,760 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,761 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,761 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:25,766 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,766 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,785 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/71ca5fae67b24affa806382ee0f1b2c5 2023-04-25 09:54:25,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/c73b79ad22f9445397df291b8154768f 2023-04-25 09:54:25,866 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,870 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,870 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,870 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,870 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,870 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,870 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,871 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,871 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,872 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,872 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,874 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,876 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:25,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50e773eb[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:25,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:25,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,878 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:25,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:25,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:25,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:25,880 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,881 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:25,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:25,883 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:25,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:25,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:25,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:25,883 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:25,885 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,885 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,885 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:25,886 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:25,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:25,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:25,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,889 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:25,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:25,889 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:25,889 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:25,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,989 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:25,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:25,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:25,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:25,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:25,993 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:25,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:25,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:25,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:25,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:25,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:25,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:25,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:25,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:25,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:25,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:25,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:25,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:25,999 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:25,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:25,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:26,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472515bc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:26,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:26,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,002 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:26,002 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:26,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:26,006 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:26,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:26,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:26,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:26,006 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:26,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:26,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:26,013 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:26,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:26,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,016 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,016 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:26,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:26,017 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:26,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:26,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416526082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:26,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416526084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:26,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416526085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,117 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:26,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:26,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:26,123 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:26,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:26,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:26,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:26,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:26,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:26,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:26,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:26,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:26,126 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:26,127 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:26,127 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:26,127 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:26,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:26,128 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:26,129 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,129 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:26,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3965c7[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:26,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:26,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,132 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:26,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:26,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:26,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:26,136 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:26,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:26,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:26,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:26,136 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:26,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:26,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:26,138 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:26,139 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:26,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:26,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:26,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:26,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:26,142 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:26,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,142 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:26,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:26,142 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:26,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:26,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416526198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,202 DEBUG [Thread-922] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:26,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/a9cbf5c8089e4392b541f495f07ef613 2023-04-25 09:54:26,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/71ca5fae67b24affa806382ee0f1b2c5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/71ca5fae67b24affa806382ee0f1b2c5 2023-04-25 09:54:26,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/71ca5fae67b24affa806382ee0f1b2c5, entries=250, sequenceid=238, filesize=38.9 K 2023-04-25 09:54:26,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/c73b79ad22f9445397df291b8154768f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/c73b79ad22f9445397df291b8154768f 2023-04-25 09:54:26,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/c73b79ad22f9445397df291b8154768f, entries=150, sequenceid=238, filesize=11.8 K 2023-04-25 09:54:26,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/a9cbf5c8089e4392b541f495f07ef613 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a9cbf5c8089e4392b541f495f07ef613 2023-04-25 09:54:26,226 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a9cbf5c8089e4392b541f495f07ef613, entries=150, sequenceid=238, filesize=11.8 K 2023-04-25 09:54:26,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~425.88 KB/436096, currentSize=33.54 KB/34350 for 5687fcb9918272e953b19ef4123710b6 in 1268ms, sequenceid=238, compaction requested=true 2023-04-25 09:54:26,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:26,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:26,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:26,227 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:54:26,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:26,227 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:54:26,230 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:54:26,231 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 164644 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:54:26,231 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:26,231 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:26,231 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:26,231 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:26,231 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/e9611b555890443abeeb4d36b580fcab, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1833e4c727dd432f9004c51940e8bcc8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2035c8ecf3be4f6d8e50bc78da22ad4b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/7d33c142fab84810a9d2e401bc10626e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/c73b79ad22f9445397df291b8154768f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=59.0 K 2023-04-25 09:54:26,231 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/683ff37db2874d119d4335d18e0e393b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/6b0fdfe64a86432290cdc115d3ad0aea, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/505eba8f6ae143658468a02daee4c4af, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/5352c271c2af4be0b751bda610cc661f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/71ca5fae67b24affa806382ee0f1b2c5] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=160.8 K 2023-04-25 09:54:26,231 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e9611b555890443abeeb4d36b580fcab, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682416461526 2023-04-25 09:54:26,231 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 683ff37db2874d119d4335d18e0e393b, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682416461526 2023-04-25 09:54:26,232 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1833e4c727dd432f9004c51940e8bcc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682416461850 2023-04-25 09:54:26,232 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6b0fdfe64a86432290cdc115d3ad0aea, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682416461847 2023-04-25 09:54:26,232 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2035c8ecf3be4f6d8e50bc78da22ad4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682416462032 2023-04-25 09:54:26,232 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 505eba8f6ae143658468a02daee4c4af, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682416462032 2023-04-25 09:54:26,233 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7d33c142fab84810a9d2e401bc10626e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682416462666 2023-04-25 09:54:26,233 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5352c271c2af4be0b751bda610cc661f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682416462666 2023-04-25 09:54:26,233 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c73b79ad22f9445397df291b8154768f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682416464846 2023-04-25 09:54:26,233 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 71ca5fae67b24affa806382ee0f1b2c5, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682416464846 2023-04-25 09:54:26,242 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:26,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:26,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:26,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:26,246 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:26,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:26,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:26,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:26,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,247 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#157 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:26,249 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:26,249 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:26,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:26,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:26,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:26,251 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#156 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:26,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:26,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:26,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:26,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:26,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:26,270 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:26,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:26,678 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/9c30d67de41c41919ba68e0f4d938b75 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/9c30d67de41c41919ba68e0f4d938b75 2023-04-25 09:54:26,690 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into 9c30d67de41c41919ba68e0f4d938b75(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:26,691 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:26,691 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=11, startTime=1682416466227; duration=0sec 2023-04-25 09:54:26,691 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:26,691 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:54:26,694 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:54:26,694 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:26,694 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:26,694 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8451102c859f433991c3ec4f347ed22a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8c8fa2c315f640fa98ca67e6fc80da0a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/6c50f56c426d45329c54be0d405a9660, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c101d5933bd74117b764384554201e94, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a9cbf5c8089e4392b541f495f07ef613] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=59.0 K 2023-04-25 09:54:26,694 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8451102c859f433991c3ec4f347ed22a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1682416461526 2023-04-25 09:54:26,695 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8c8fa2c315f640fa98ca67e6fc80da0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1682416461850 2023-04-25 09:54:26,695 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6c50f56c426d45329c54be0d405a9660, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1682416462032 2023-04-25 09:54:26,696 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c101d5933bd74117b764384554201e94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682416462666 2023-04-25 09:54:26,696 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425fc89561b4ac643198709e201d85ef6c1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425fc89561b4ac643198709e201d85ef6c1 2023-04-25 09:54:26,696 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a9cbf5c8089e4392b541f495f07ef613, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682416464846 2023-04-25 09:54:26,708 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#159 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:26,729 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/8678d733cb2f452fad64a2a71137f4bb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8678d733cb2f452fad64a2a71137f4bb 2023-04-25 09:54:26,735 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into 8678d733cb2f452fad64a2a71137f4bb(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:26,735 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:26,735 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=11, startTime=1682416466227; duration=0sec 2023-04-25 09:54:26,735 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:26,738 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/08ea4a6c84c84c3d8bd11c08cd70ef72 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/08ea4a6c84c84c3d8bd11c08cd70ef72 2023-04-25 09:54:26,744 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 08ea4a6c84c84c3d8bd11c08cd70ef72(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:26,744 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:26,744 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=11, startTime=1682416466227; duration=0sec 2023-04-25 09:54:26,744 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:26,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:26,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:26,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416466855, max timestamp=1682416466871Num uniques 859; 2023-04-25 09:54:26,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:26,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416466855, max timestamp=1682416466871Num uniques 859; 2023-04-25 09:54:26,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416466855, max timestamp=1682416466871Num uniques 859; 2023-04-25 09:54:26,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:26,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:26,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416466900, max timestamp=1682416466909Num uniques 1068; 2023-04-25 09:54:26,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-25 09:54:26,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416466900, max timestamp=1682416466909Num uniques 1068; 2023-04-25 09:54:26,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416466900, max timestamp=1682416466909Num uniques 1068; 2023-04-25 09:54:26,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-25 09:54:26,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-25 09:54:26,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682416466950, max timestamp=1682416466960Num uniques 1077; 2023-04-25 09:54:26,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-25 09:54:26,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682416466950, max timestamp=1682416466960Num uniques 1077; 2023-04-25 09:54:26,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-25 09:54:26,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682416466950, max timestamp=1682416466960Num uniques 1077; 2023-04-25 09:54:26,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-25 09:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682416466987, max timestamp=1682416467009Num uniques 1086; 2023-04-25 09:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 09:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682416466987, max timestamp=1682416467009Num uniques 1086; 2023-04-25 09:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 09:54:27,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682416466987, max timestamp=1682416467009Num uniques 1086; 2023-04-25 09:54:27,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-25 09:54:27,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416467028, max timestamp=1682416467055Num uniques 1095; 2023-04-25 09:54:27,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 09:54:27,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416467028, max timestamp=1682416467055Num uniques 1095; 2023-04-25 09:54:27,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416467028, max timestamp=1682416467055Num uniques 1095; 2023-04-25 09:54:27,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 09:54:27,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 09:54:27,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682416467072, max timestamp=1682416467092Num uniques 1054; 2023-04-25 09:54:27,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-25 09:54:27,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682416467072, max timestamp=1682416467092Num uniques 1054; 2023-04-25 09:54:27,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-25 09:54:27,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682416467072, max timestamp=1682416467092Num uniques 1054; 2023-04-25 09:54:27,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-25 09:54:27,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44076 deadline: 1682416527108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44090 deadline: 1682416527109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416527110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416527110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,121 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=246, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/2ab59f0516af440f96960235c45ce954 2023-04-25 09:54:27,131 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/a2dd003cc8174d67bed3251968258e1c 2023-04-25 09:54:27,141 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/942b160c59264e3e930e19aebf09a8b4 2023-04-25 09:54:27,145 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/2ab59f0516af440f96960235c45ce954 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ab59f0516af440f96960235c45ce954 2023-04-25 09:54:27,149 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ab59f0516af440f96960235c45ce954, entries=150, sequenceid=246, filesize=25.3 K 2023-04-25 09:54:27,150 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/a2dd003cc8174d67bed3251968258e1c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/a2dd003cc8174d67bed3251968258e1c 2023-04-25 09:54:27,158 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/a2dd003cc8174d67bed3251968258e1c, entries=150, sequenceid=246, filesize=11.8 K 2023-04-25 09:54:27,159 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/942b160c59264e3e930e19aebf09a8b4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/942b160c59264e3e930e19aebf09a8b4 2023-04-25 09:54:27,163 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/942b160c59264e3e930e19aebf09a8b4, entries=150, sequenceid=246, filesize=11.8 K 2023-04-25 09:54:27,164 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 5687fcb9918272e953b19ef4123710b6 in 894ms, sequenceid=246, compaction requested=false 2023-04-25 09:54:27,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:27,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,166 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@158a37f7[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,174 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,174 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,180 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,180 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:27,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682416467108, max timestamp=1682416467214Num uniques 1113; 2023-04-25 09:54:27,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-25 09:54:27,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=118.53 KB heapSize=332.91 KB 2023-04-25 09:54:27,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682416467108, max timestamp=1682416467214Num uniques 1113; 2023-04-25 09:54:27,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-25 09:54:27,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:27,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682416467108, max timestamp=1682416467214Num uniques 1113; 2023-04-25 09:54:27,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:27,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-25 09:54:27,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:27,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:27,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:27,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:27,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416467227, max timestamp=1682416467234Num uniques 859; 2023-04-25 09:54:27,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416467227, max timestamp=1682416467234Num uniques 859; 2023-04-25 09:54:27,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:54:27,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:54:27,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416467227, max timestamp=1682416467234Num uniques 859; 2023-04-25 09:54:27,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:54:27,237 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d8c6e0c7ac744d37a16320a4144f7ecd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d8c6e0c7ac744d37a16320a4144f7ecd 2023-04-25 09:54:27,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416527250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416527253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416527254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416527255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,281 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:27,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:27,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:27,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:27,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:27,284 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:27,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:27,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:27,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:27,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:27,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:27,287 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,287 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:27,287 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:27,287 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e196686[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,305 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,310 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,311 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,311 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416527356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416527357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416527357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416527357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,411 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:27,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:27,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:27,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:27,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:27,414 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:27,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:27,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:27,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,416 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:27,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:27,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:27,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:27,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:27,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:27,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,423 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fe040c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,426 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,426 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,436 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,436 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,438 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,439 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,442 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,442 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:27,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,542 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:27,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:27,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:27,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:27,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:27,545 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:27,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:27,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:27,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:27,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:27,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:27,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:27,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:27,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,554 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40385ca9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,557 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,560 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,560 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416527562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44106 deadline: 1682416527562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416527563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416527563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,564 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:27,568 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,568 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,643 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/acc32421ca764d65bb8039ef465f8bb4 2023-04-25 09:54:27,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/1504af33a12f47f0828b00a46fb5e684 2023-04-25 09:54:27,668 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:27,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:27,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:27,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:27,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:27,671 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:27,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:27,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:27,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,673 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:27,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:27,674 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,675 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:27,675 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:27,675 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,676 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449eed6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,678 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,678 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,681 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,684 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,684 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,692 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:27,692 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,806 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:27,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:27,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:27,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:27,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:27,809 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:27,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:27,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:27,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:27,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:27,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:27,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:27,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,815 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3151958f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,817 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,819 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,821 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,821 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,827 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,827 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416527864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416527864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416527865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416527865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,928 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:27,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:27,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:27,931 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:27,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:27,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:27,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,932 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,933 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,933 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:27,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:27,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:27,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:27,934 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,934 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:27,935 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:27,935 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:27,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:27,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ee29cd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:27,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:27,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:27,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:27,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:27,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:27,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:27,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:27,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:27,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,946 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,948 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:27,965 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:27,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:27,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:27,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:27,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:27,965 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:27,965 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:27,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:27,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:28,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/57bae6260c11449f9d30b141ef961bed 2023-04-25 09:54:28,066 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:28,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:28,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/acc32421ca764d65bb8039ef465f8bb4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/acc32421ca764d65bb8039ef465f8bb4 2023-04-25 09:54:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:28,069 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:28,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:28,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:28,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:28,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:28,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:28,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/acc32421ca764d65bb8039ef465f8bb4, entries=150, sequenceid=375, filesize=25.4 K 2023-04-25 09:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:28,076 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,076 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:28,076 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:28,076 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:28,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/1504af33a12f47f0828b00a46fb5e684 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1504af33a12f47f0828b00a46fb5e684 2023-04-25 09:54:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,079 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b1ea6d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:28,082 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:28,082 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:28,083 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,083 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,084 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:28,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:28,088 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:28,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:28,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:28,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1504af33a12f47f0828b00a46fb5e684, entries=150, sequenceid=375, filesize=11.9 K 2023-04-25 09:54:28,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:28,088 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:28,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/57bae6260c11449f9d30b141ef961bed as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/57bae6260c11449f9d30b141ef961bed 2023-04-25 09:54:28,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,092 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/57bae6260c11449f9d30b141ef961bed, entries=150, sequenceid=375, filesize=11.9 K 2023-04-25 09:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~105.66 KB/108192, currentSize=147.60 KB/151140 for 5687fcb9918272e953b19ef4123710b6 in 878ms, sequenceid=375, compaction requested=true 2023-04-25 09:54:28,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:28,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:28,095 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:28,095 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:28,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:28,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:28,096 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78214 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:28,096 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36634 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:28,096 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:28,096 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:28,096 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,096 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8678d733cb2f452fad64a2a71137f4bb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ab59f0516af440f96960235c45ce954, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/acc32421ca764d65bb8039ef465f8bb4] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=76.4 K 2023-04-25 09:54:28,096 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,096 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/9c30d67de41c41919ba68e0f4d938b75, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/a2dd003cc8174d67bed3251968258e1c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1504af33a12f47f0828b00a46fb5e684] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=35.8 K 2023-04-25 09:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,097 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8678d733cb2f452fad64a2a71137f4bb, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682416464846 2023-04-25 09:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,100 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9c30d67de41c41919ba68e0f4d938b75, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682416464846 2023-04-25 09:54:28,100 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2ab59f0516af440f96960235c45ce954, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416464960 2023-04-25 09:54:28,100 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a2dd003cc8174d67bed3251968258e1c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416464960 2023-04-25 09:54:28,101 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting acc32421ca764d65bb8039ef465f8bb4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682416467214 2023-04-25 09:54:28,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,101 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1504af33a12f47f0828b00a46fb5e684, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682416467214 2023-04-25 09:54:28,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:28,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:28,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:28,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,102 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:28,103 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:28,111 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#166 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:28,125 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#165 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:28,203 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:28,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:28,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:28,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:28,207 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:28,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:28,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:28,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,208 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,208 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:28,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:28,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:28,211 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-25 09:54:28,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:28,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:28,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:28,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:28,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:28,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:28,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:28,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:28,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416528374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416528375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416528375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416528379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416528480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416528480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416528480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416528481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,528 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/aed15c2932a540b18a8cc55166de8245 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/aed15c2932a540b18a8cc55166de8245 2023-04-25 09:54:28,533 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into aed15c2932a540b18a8cc55166de8245(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:28,533 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:28,533 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=13, startTime=1682416468095; duration=0sec 2023-04-25 09:54:28,533 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:28,533 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:28,534 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36634 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:28,534 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:28,534 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,535 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/08ea4a6c84c84c3d8bd11c08cd70ef72, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/942b160c59264e3e930e19aebf09a8b4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/57bae6260c11449f9d30b141ef961bed] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=35.8 K 2023-04-25 09:54:28,535 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 08ea4a6c84c84c3d8bd11c08cd70ef72, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682416464846 2023-04-25 09:54:28,535 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 942b160c59264e3e930e19aebf09a8b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1682416464960 2023-04-25 09:54:28,536 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 57bae6260c11449f9d30b141ef961bed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682416467214 2023-04-25 09:54:28,541 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#168 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:28,627 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425926b931890c44cf8ab2780f43280e2d2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425926b931890c44cf8ab2780f43280e2d2 2023-04-25 09:54:28,632 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=411, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/11dbc147aaa54c7597b9fa90abcc78a0 2023-04-25 09:54:28,645 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/f4d89abd5e3049869816172582f11006 2023-04-25 09:54:28,654 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/a746622f23db4b40b58a9baeafdd028c 2023-04-25 09:54:28,658 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/11dbc147aaa54c7597b9fa90abcc78a0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/11dbc147aaa54c7597b9fa90abcc78a0 2023-04-25 09:54:28,662 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/11dbc147aaa54c7597b9fa90abcc78a0, entries=150, sequenceid=411, filesize=25.4 K 2023-04-25 09:54:28,663 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/f4d89abd5e3049869816172582f11006 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f4d89abd5e3049869816172582f11006 2023-04-25 09:54:28,667 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f4d89abd5e3049869816172582f11006, entries=150, sequenceid=411, filesize=11.9 K 2023-04-25 09:54:28,668 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/a746622f23db4b40b58a9baeafdd028c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a746622f23db4b40b58a9baeafdd028c 2023-04-25 09:54:28,672 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a746622f23db4b40b58a9baeafdd028c, entries=150, sequenceid=411, filesize=11.9 K 2023-04-25 09:54:28,673 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 5687fcb9918272e953b19ef4123710b6 in 462ms, sequenceid=411, compaction requested=false 2023-04-25 09:54:28,673 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:28,673 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:28,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,677 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:28,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c591b3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:28,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:28,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,681 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:28,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:28,682 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:28,685 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:28,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:28,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:28,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:28,685 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:28,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:28,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:54:28,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:28,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:28,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:28,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:28,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:28,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:28,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,703 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425215fe8f4de244a878e8d23ad5a9f0485 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425215fe8f4de244a878e8d23ad5a9f0485 2023-04-25 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:28,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:28,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,708 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,708 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416528731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416528731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416528732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416528733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,808 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:28,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:28,813 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:28,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:28,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:28,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,814 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:28,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:28,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:28,816 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,816 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:28,816 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:28,816 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:28,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,818 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:28,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248592e9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:28,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:28,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:28,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:28,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:28,828 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:28,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:28,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:28,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:28,829 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:28,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:28,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:28,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,834 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:28,835 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:28,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416528834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416528835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416528837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416528837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,935 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:28,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:28,938 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:28,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:28,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:28,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,951 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/437684168133401884c9587a1340437b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/437684168133401884c9587a1340437b 2023-04-25 09:54:28,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:28,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:28,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:28,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:28,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:28,956 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 437684168133401884c9587a1340437b(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:28,956 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:28,957 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=13, startTime=1682416468095; duration=0sec 2023-04-25 09:54:28,957 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:28,961 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,961 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:28,961 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:28,962 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:28,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:28,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:28,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:28,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:28,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50477f5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:28,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:28,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:28,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,969 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/e6a6d3179ae746da8ea7590d688d2d0b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e6a6d3179ae746da8ea7590d688d2d0b 2023-04-25 09:54:28,969 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,969 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:28,969 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:28,973 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:28,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:28,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:28,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:28,973 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:28,974 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into e6a6d3179ae746da8ea7590d688d2d0b(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:28,974 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:28,974 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=13, startTime=1682416468095; duration=0sec 2023-04-25 09:54:28,974 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:28,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:28,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:28,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:28,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:28,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:28,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:28,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:28,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:28,985 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:28,985 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:28,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,986 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:28,986 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416529039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416529039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416529040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416529042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,086 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:29,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:29,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:29,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:29,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:29,089 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:29,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:29,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:29,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:29,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:29,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:29,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:29,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:29,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:29,092 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:29,092 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:29,092 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:29,093 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:29,094 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bb17ba[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:29,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:29,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:29,101 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,101 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:29,103 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:29,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:29,105 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:29,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:29,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:29,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:29,106 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:29,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:29,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:29,108 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:29,108 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/ab90d9dd5bac4796acbe8424c3eadcf6 2023-04-25 09:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:29,110 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:29,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:29,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:29,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:29,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:29,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:29,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:29,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:29,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:29,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,114 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:29,114 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:29,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/3330ad00429e4fa98fc134a3e23b2e87 2023-04-25 09:54:29,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/00858272fb754769931624ef76214306 2023-04-25 09:54:29,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/ab90d9dd5bac4796acbe8424c3eadcf6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/ab90d9dd5bac4796acbe8424c3eadcf6 2023-04-25 09:54:29,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/ab90d9dd5bac4796acbe8424c3eadcf6, entries=250, sequenceid=425, filesize=39.2 K 2023-04-25 09:54:29,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/3330ad00429e4fa98fc134a3e23b2e87 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/3330ad00429e4fa98fc134a3e23b2e87 2023-04-25 09:54:29,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/3330ad00429e4fa98fc134a3e23b2e87, entries=150, sequenceid=425, filesize=11.9 K 2023-04-25 09:54:29,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/00858272fb754769931624ef76214306 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/00858272fb754769931624ef76214306 2023-04-25 09:54:29,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/00858272fb754769931624ef76214306, entries=150, sequenceid=425, filesize=11.9 K 2023-04-25 09:54:29,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5687fcb9918272e953b19ef4123710b6 in 467ms, sequenceid=425, compaction requested=true 2023-04-25 09:54:29,154 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:29,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:29,154 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:29,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:29,154 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:29,156 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:29,156 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92700 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:29,156 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:29,156 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:29,156 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:29,156 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:29,156 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/aed15c2932a540b18a8cc55166de8245, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f4d89abd5e3049869816172582f11006, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/3330ad00429e4fa98fc134a3e23b2e87] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=36.2 K 2023-04-25 09:54:29,156 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e6a6d3179ae746da8ea7590d688d2d0b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/11dbc147aaa54c7597b9fa90abcc78a0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/ab90d9dd5bac4796acbe8424c3eadcf6] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=90.5 K 2023-04-25 09:54:29,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:29,157 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting aed15c2932a540b18a8cc55166de8245, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682416467214 2023-04-25 09:54:29,157 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e6a6d3179ae746da8ea7590d688d2d0b, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682416467214 2023-04-25 09:54:29,157 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f4d89abd5e3049869816172582f11006, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1682416467248 2023-04-25 09:54:29,158 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 11dbc147aaa54c7597b9fa90abcc78a0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1682416467248 2023-04-25 09:54:29,158 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3330ad00429e4fa98fc134a3e23b2e87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682416468686 2023-04-25 09:54:29,159 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ab90d9dd5bac4796acbe8424c3eadcf6, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682416468378 2023-04-25 09:54:29,179 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#175 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:29,185 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#174 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:29,214 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:29,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:29,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:29,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:29,223 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:29,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:29,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:29,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:29,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:29,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:29,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:29,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:29,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:29,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:29,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:29,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:29,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:29,232 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:29,232 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:29,232 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:54:29,232 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:29,232 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:29,233 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:29,233 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:29,233 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:29,233 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:29,252 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/d7d3c4d76fc84a7cbcdc0502507d4ea0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/d7d3c4d76fc84a7cbcdc0502507d4ea0 2023-04-25 09:54:29,255 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425724623578b524f868317469e8024bd6d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425724623578b524f868317469e8024bd6d 2023-04-25 09:54:29,257 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into d7d3c4d76fc84a7cbcdc0502507d4ea0(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:29,257 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:29,257 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=13, startTime=1682416469154; duration=0sec 2023-04-25 09:54:29,257 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:29,257 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:29,258 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:29,258 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:29,258 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:29,258 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/437684168133401884c9587a1340437b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a746622f23db4b40b58a9baeafdd028c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/00858272fb754769931624ef76214306] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=36.2 K 2023-04-25 09:54:29,259 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 437684168133401884c9587a1340437b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1682416467214 2023-04-25 09:54:29,259 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a746622f23db4b40b58a9baeafdd028c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1682416467248 2023-04-25 09:54:29,260 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 00858272fb754769931624ef76214306, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682416468686 2023-04-25 09:54:29,281 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#177 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:29,304 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/c37d345fd4e54d65aa8159765d696633 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c37d345fd4e54d65aa8159765d696633 2023-04-25 09:54:29,311 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into c37d345fd4e54d65aa8159765d696633(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:29,311 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:29,311 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=13, startTime=1682416469156; duration=0sec 2023-04-25 09:54:29,311 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:29,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:29,345 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:29,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416529351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416529351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416529353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416529354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416529455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416529455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416529457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416529459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,602 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/f71ac8ded8504c5c893fa831237cf905 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f71ac8ded8504c5c893fa831237cf905 2023-04-25 09:54:29,607 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into f71ac8ded8504c5c893fa831237cf905(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:29,607 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:29,607 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=13, startTime=1682416469154; duration=0sec 2023-04-25 09:54:29,607 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:29,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416529657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416529658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416529661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416529663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,685 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/2bb4d89a8c5a4c689c9e6507b412ede4 2023-04-25 09:54:29,720 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/660d03f186004556bc875c67abbce58b 2023-04-25 09:54:29,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416529960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416529962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416529965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:29,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:29,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416529967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,140 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/f94d2fadc6824dadb92923d485dcee57 2023-04-25 09:54:30,146 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/2bb4d89a8c5a4c689c9e6507b412ede4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2bb4d89a8c5a4c689c9e6507b412ede4 2023-04-25 09:54:30,150 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2bb4d89a8c5a4c689c9e6507b412ede4, entries=150, sequenceid=446, filesize=25.4 K 2023-04-25 09:54:30,152 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/660d03f186004556bc875c67abbce58b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/660d03f186004556bc875c67abbce58b 2023-04-25 09:54:30,159 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/660d03f186004556bc875c67abbce58b, entries=150, sequenceid=446, filesize=11.9 K 2023-04-25 09:54:30,161 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/f94d2fadc6824dadb92923d485dcee57 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/f94d2fadc6824dadb92923d485dcee57 2023-04-25 09:54:30,170 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/f94d2fadc6824dadb92923d485dcee57, entries=150, sequenceid=446, filesize=11.9 K 2023-04-25 09:54:30,171 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5687fcb9918272e953b19ef4123710b6 in 939ms, sequenceid=446, compaction requested=false 2023-04-25 09:54:30,171 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:30,171 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,173 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d465d17[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,181 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,181 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,183 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,187 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,188 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,188 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,188 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:30,188 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,188 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,188 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:30,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:54:30,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:30,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:30,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:30,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:30,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:30,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:30,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416530259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,288 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:30,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:30,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:30,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:30,292 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:30,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:30,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:30,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:30,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:30,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:30,296 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,296 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:30,296 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:30,296 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@655c888c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,302 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,302 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,303 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,307 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,307 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:30,312 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,312 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416530363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,412 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:30,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:30,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:30,415 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:30,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:30,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:30,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:30,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:30,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:30,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:30,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:30,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:30,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:30,421 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c1f00d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,426 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,426 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,427 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,430 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,430 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,439 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,439 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,439 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,439 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,446 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,446 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416530464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416530467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416530471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416530471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,546 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:30,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:30,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:30,549 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:30,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:30,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:30,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,551 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:30,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:30,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:30,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:30,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:30,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:30,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,555 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43472908[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,557 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,562 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,562 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416530567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,571 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,572 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,575 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,576 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,576 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,647 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425de7df2155409443faa46515d7fc96b9e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425de7df2155409443faa46515d7fc96b9e 2023-04-25 09:54:30,676 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:30,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:30,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:30,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:30,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:30,679 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:30,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:30,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:30,680 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,680 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,681 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:30,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:30,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:30,683 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,683 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:30,683 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:30,683 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c366a92[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,693 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,693 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,697 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,699 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,699 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,700 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,799 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:30,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:30,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:30,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:30,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:30,803 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:30,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:30,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:30,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,805 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:30,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:30,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:30,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:30,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:30,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:30,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d2f8970[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,814 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,816 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,816 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:30,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,831 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,831 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:30,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416530870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,931 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:30,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:30,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:30,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:30,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:30,935 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:30,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:30,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:30,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,937 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:30,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:30,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:30,939 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,939 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:30,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:30,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:30,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:30,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:30,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:30,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:30,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:30,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a24340[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:30,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:30,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:30,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:30,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:30,947 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:30,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:30,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:30,949 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:30,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:30,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:30,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:30,949 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:30,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:30,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:30,951 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:30,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:30,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:30,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:30,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,956 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:30,956 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:30,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:30,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:30,956 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:30,956 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:30,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:30,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:31,056 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:31,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:31,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:31,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:31,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:31,059 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:31,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:31,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:31,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,062 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:31,064 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=463, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/94e0f75211f04baab3316835ebd5a44f 2023-04-25 09:54:31,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:31,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:31,064 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,064 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:31,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:31,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:31,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:31,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd72f15[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:31,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:31,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,075 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:31,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:31,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:31,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:31,079 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:31,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:31,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:31,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:31,079 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:31,082 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,082 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,082 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,083 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:31,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,089 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:31,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,089 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:31,189 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:31,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:31,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:31,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:31,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:31,192 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:31,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:31,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:31,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:31,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:31,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:31,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:31,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:31,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:31,196 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,196 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:31,196 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:31,196 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:31,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:31,198 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:31,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ab207[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:31,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:31,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:31,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:31,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:31,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:31,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:31,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:31,206 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:31,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:31,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:31,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:31,206 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:31,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,208 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,230 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:31,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,230 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:31,230 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:31,331 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:31,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:31,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:31,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:31,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:31,334 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:31,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:31,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:31,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:31,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:31,338 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,338 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:31,338 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:31,338 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,340 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:31,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651fdfed[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:31,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:31,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,344 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:31,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:31,345 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:31,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:31,348 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:31,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:31,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:31,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:31,348 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:31,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:31,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:31,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,354 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:31,354 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:31,354 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:31,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416531377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,454 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:31,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:31,457 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:31,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:31,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:31,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:31,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:31,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:31,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:31,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:31,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:31,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:31,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26e4384d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:31,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:31,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,467 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:31,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:31,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:31,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:31,471 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:31,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:31,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:31,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:31,472 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:31,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:31,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416531473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:31,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416531475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:31,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/71c6f029c5f84cd69eb075160ded1874 2023-04-25 09:54:31,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416531476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:31,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:31,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:31,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:31,481 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:31,482 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:31,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:31,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:31,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:31,482 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:31,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416531480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/afb3facb4bac4d47b624225e6f1e260e 2023-04-25 09:54:31,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/94e0f75211f04baab3316835ebd5a44f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/94e0f75211f04baab3316835ebd5a44f 2023-04-25 09:54:31,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/94e0f75211f04baab3316835ebd5a44f, entries=200, sequenceid=463, filesize=32.3 K 2023-04-25 09:54:31,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/71c6f029c5f84cd69eb075160ded1874 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/71c6f029c5f84cd69eb075160ded1874 2023-04-25 09:54:31,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/71c6f029c5f84cd69eb075160ded1874, entries=150, sequenceid=463, filesize=11.9 K 2023-04-25 09:54:31,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/afb3facb4bac4d47b624225e6f1e260e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/afb3facb4bac4d47b624225e6f1e260e 2023-04-25 09:54:31,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/afb3facb4bac4d47b624225e6f1e260e, entries=150, sequenceid=463, filesize=11.9 K 2023-04-25 09:54:31,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5687fcb9918272e953b19ef4123710b6 in 1286ms, sequenceid=463, compaction requested=true 2023-04-25 09:54:31,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:31,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:31,506 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:31,506 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:31,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:31,506 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:31,507 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:31,507 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:31,507 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:31,507 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:31,508 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,508 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,508 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f71ac8ded8504c5c893fa831237cf905, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/660d03f186004556bc875c67abbce58b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/71c6f029c5f84cd69eb075160ded1874] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=36.3 K 2023-04-25 09:54:31,508 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/d7d3c4d76fc84a7cbcdc0502507d4ea0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2bb4d89a8c5a4c689c9e6507b412ede4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/94e0f75211f04baab3316835ebd5a44f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=83.7 K 2023-04-25 09:54:31,508 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f71ac8ded8504c5c893fa831237cf905, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682416468686 2023-04-25 09:54:31,508 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting d7d3c4d76fc84a7cbcdc0502507d4ea0, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682416468686 2023-04-25 09:54:31,509 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 660d03f186004556bc875c67abbce58b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682416468725 2023-04-25 09:54:31,509 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2bb4d89a8c5a4c689c9e6507b412ede4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682416468725 2023-04-25 09:54:31,509 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 94e0f75211f04baab3316835ebd5a44f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682416469350 2023-04-25 09:54:31,509 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 71c6f029c5f84cd69eb075160ded1874, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682416469350 2023-04-25 09:54:31,521 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#184 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:31,525 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#183 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:31,582 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:31,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:31,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:31,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:31,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:31,585 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:31,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:31,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:31,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,587 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:31,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:31,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:31,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:31,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:31,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:31,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:31,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:31,589 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:31,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:31,938 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/335538b44466463682fa75969601bafa as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/335538b44466463682fa75969601bafa 2023-04-25 09:54:31,944 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into 335538b44466463682fa75969601bafa(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:31,945 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:31,945 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=13, startTime=1682416471506; duration=0sec 2023-04-25 09:54:31,945 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:31,945 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:31,946 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:31,946 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:31,946 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:31,946 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c37d345fd4e54d65aa8159765d696633, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/f94d2fadc6824dadb92923d485dcee57, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/afb3facb4bac4d47b624225e6f1e260e] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=36.3 K 2023-04-25 09:54:31,947 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c37d345fd4e54d65aa8159765d696633, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682416468686 2023-04-25 09:54:31,947 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f94d2fadc6824dadb92923d485dcee57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1682416468725 2023-04-25 09:54:31,947 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting afb3facb4bac4d47b624225e6f1e260e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682416469350 2023-04-25 09:54:31,953 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:31,962 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/5d10e10e9f5a43deaad3859b6c2e778e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5d10e10e9f5a43deaad3859b6c2e778e 2023-04-25 09:54:31,967 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 5d10e10e9f5a43deaad3859b6c2e778e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:31,967 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:31,967 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=13, startTime=1682416471506; duration=0sec 2023-04-25 09:54:31,967 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:32,007 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425b6c320ea587d495787801df4084207c2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b6c320ea587d495787801df4084207c2 2023-04-25 09:54:32,013 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=481, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/fc82061188d344bdb45a8f6cddba000e 2023-04-25 09:54:32,347 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/e02d956d6cac4cdf8b1306cfc36f57be as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e02d956d6cac4cdf8b1306cfc36f57be 2023-04-25 09:54:32,352 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into e02d956d6cac4cdf8b1306cfc36f57be(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:32,352 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:32,352 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=13, startTime=1682416471506; duration=0sec 2023-04-25 09:54:32,353 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:32,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:32,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:32,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44112 deadline: 1682416532404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,424 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/07b83fa464434c37b3d811141a49243e 2023-04-25 09:54:32,434 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/0037a6d972a84b178c09a78bbb195bb0 2023-04-25 09:54:32,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/fc82061188d344bdb45a8f6cddba000e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/fc82061188d344bdb45a8f6cddba000e 2023-04-25 09:54:32,443 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/fc82061188d344bdb45a8f6cddba000e, entries=150, sequenceid=481, filesize=25.4 K 2023-04-25 09:54:32,445 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/07b83fa464434c37b3d811141a49243e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/07b83fa464434c37b3d811141a49243e 2023-04-25 09:54:32,449 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/07b83fa464434c37b3d811141a49243e, entries=150, sequenceid=481, filesize=11.9 K 2023-04-25 09:54:32,450 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/0037a6d972a84b178c09a78bbb195bb0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/0037a6d972a84b178c09a78bbb195bb0 2023-04-25 09:54:32,454 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/0037a6d972a84b178c09a78bbb195bb0, entries=150, sequenceid=481, filesize=11.9 K 2023-04-25 09:54:32,455 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5687fcb9918272e953b19ef4123710b6 in 866ms, sequenceid=481, compaction requested=false 2023-04-25 09:54:32,455 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:32,455 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,455 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:32,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:32,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:32,456 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:32,457 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,457 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,458 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,458 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:32,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d90403[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:32,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:32,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,461 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:32,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:32,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:32,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:32,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:32,463 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:32,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:32,466 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:32,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:32,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:32,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:32,466 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:32,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,470 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:32,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,474 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:32,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,474 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:32,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:32,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:54:32,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:32,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:32,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:32,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:32,508 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:32,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:32,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:32,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416532558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,574 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:32,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:32,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:32,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:32,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:32,577 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:32,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:32,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:32,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,580 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,581 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,581 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:32,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:32,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:32,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:32,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:32,582 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:32,582 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:32,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:32,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65028215[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:32,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:32,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,589 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:32,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:32,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:32,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:32,593 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:32,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:32,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:32,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:32,593 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:32,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,601 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,604 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:32,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:32,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,605 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:32,605 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:32,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:32,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:32,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416532662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,705 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:32,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:32,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:32,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:32,708 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:32,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:32,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:32,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,710 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,710 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:32,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:32,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:32,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:32,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:32,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:32,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:32,714 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,714 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:32,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef9c9ef[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:32,716 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:32,716 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,718 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:32,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:32,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:32,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:32,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:32,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:32,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:32,722 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:32,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:32,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:32,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:32,722 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:32,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,724 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,726 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:32,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:32,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,728 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,729 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:32,729 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:32,729 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:32,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,829 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:32,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:32,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:32,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:32,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:32,833 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:32,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:32,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:32,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,836 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,836 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,837 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,837 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:32,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:32,838 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,838 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:32,838 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:32,838 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,840 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7958a02d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:32,842 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:32,842 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,843 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:32,843 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:32,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:32,844 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:32,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:32,847 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:32,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:32,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:32,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:32,847 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:32,849 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,849 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,849 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,853 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:32,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:32,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:32,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,856 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,856 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,856 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,856 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:32,856 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:32,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:32,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416532865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,933 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042533b9a67e8c5d4eec9a6cb4c2557a17b2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042533b9a67e8c5d4eec9a6cb4c2557a17b2 2023-04-25 09:54:32,956 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:32,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:32,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:32,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:32,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:32,960 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:32,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:32,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:32,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:32,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:32,963 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,963 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:32,964 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:32,964 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:32,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d4a19[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:32,967 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:32,967 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,972 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:32,972 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,972 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:32,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:32,974 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,974 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:32,976 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:32,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:32,976 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:32,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:32,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:32,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:32,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:32,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:32,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:32,983 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:32,983 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:32,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,983 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:32,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:32,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,083 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,087 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:33,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:33,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:33,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:33,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74be628d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:33,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,095 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,095 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:33,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:33,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:33,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:33,100 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:33,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:33,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:33,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:33,100 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:33,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,103 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:33,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:33,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,106 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:33,106 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:33,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416533169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,206 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,209 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,213 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,213 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,213 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:33,213 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:33,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:33,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:33,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:33,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44386b44[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:33,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,218 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:33,218 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,218 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:33,220 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:33,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:33,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:33,222 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:33,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:33,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:33,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:33,222 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:33,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,229 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:33,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:33,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:33,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,230 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:33,230 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,330 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,333 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,334 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:33,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:33,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:33,338 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:33,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:33,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ddf226[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:33,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,340 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=499, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/8ec67f2c295946e39f0ad3367d9b58a3 2023-04-25 09:54:33,341 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:33,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:33,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:33,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:33,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:33,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:33,350 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:33,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:33,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:33,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:33,351 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:33,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,354 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,357 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,358 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:33,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:33,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,358 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:33,358 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,458 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,462 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,465 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,465 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,465 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,467 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,467 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,467 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:33,467 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:33,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:33,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17bf62a6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:33,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:33,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,472 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,472 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:33,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:33,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:33,476 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:33,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:33,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:33,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:33,477 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:33,479 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,479 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,479 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,483 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:33,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,484 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:33,484 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:33,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:33,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416533486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:33,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416533487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,490 DEBUG [Thread-918] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:33,491 DEBUG [Thread-920] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:33,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:33,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416533492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:33,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416533493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,495 DEBUG [Thread-916] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:33,496 DEBUG [Thread-924] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:33,584 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,587 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:33,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:33,594 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1092d0d6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,598 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,598 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:33,598 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:33,598 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:33,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:33,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:33,602 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:33,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:33,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:33,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:33,602 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:33,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,604 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,608 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:33,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:33,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:33,608 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,609 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:33,609 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:33,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:33,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416533672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,709 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,712 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,712 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,712 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,712 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,717 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,719 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,719 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,719 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:33,719 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:33,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,721 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:33,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1777fbcb[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:33,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:33,725 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,725 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:33,728 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:33,728 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:33,728 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:33,728 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:33,728 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:33,730 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,730 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:33,730 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:33,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:33,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:33,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,738 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:33,738 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:33,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:33,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:33,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/67d395aa9c29475bbd75a25cc6b89566 2023-04-25 09:54:33,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/368ef996a3714b0bb8d9ea52beb94b57 2023-04-25 09:54:33,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/8ec67f2c295946e39f0ad3367d9b58a3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8ec67f2c295946e39f0ad3367d9b58a3 2023-04-25 09:54:33,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8ec67f2c295946e39f0ad3367d9b58a3, entries=200, sequenceid=499, filesize=32.3 K 2023-04-25 09:54:33,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/67d395aa9c29475bbd75a25cc6b89566 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/67d395aa9c29475bbd75a25cc6b89566 2023-04-25 09:54:33,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/67d395aa9c29475bbd75a25cc6b89566, entries=150, sequenceid=499, filesize=11.9 K 2023-04-25 09:54:33,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/368ef996a3714b0bb8d9ea52beb94b57 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/368ef996a3714b0bb8d9ea52beb94b57 2023-04-25 09:54:33,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/368ef996a3714b0bb8d9ea52beb94b57, entries=150, sequenceid=499, filesize=11.9 K 2023-04-25 09:54:33,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5687fcb9918272e953b19ef4123710b6 in 1288ms, sequenceid=499, compaction requested=true 2023-04-25 09:54:33,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:33,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:33,796 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:33,796 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:33,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:33,796 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:33,803 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:33,803 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:33,803 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:33,803 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:33,803 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,803 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,803 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e02d956d6cac4cdf8b1306cfc36f57be, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/fc82061188d344bdb45a8f6cddba000e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8ec67f2c295946e39f0ad3367d9b58a3] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=83.8 K 2023-04-25 09:54:33,803 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/335538b44466463682fa75969601bafa, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/07b83fa464434c37b3d811141a49243e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/67d395aa9c29475bbd75a25cc6b89566] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=36.4 K 2023-04-25 09:54:33,804 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e02d956d6cac4cdf8b1306cfc36f57be, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682416469350 2023-04-25 09:54:33,804 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 335538b44466463682fa75969601bafa, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682416469350 2023-04-25 09:54:33,804 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting fc82061188d344bdb45a8f6cddba000e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1682416470229 2023-04-25 09:54:33,804 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8ec67f2c295946e39f0ad3367d9b58a3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416472385 2023-04-25 09:54:33,805 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 07b83fa464434c37b3d811141a49243e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1682416470229 2023-04-25 09:54:33,805 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 67d395aa9c29475bbd75a25cc6b89566, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416472385 2023-04-25 09:54:33,817 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:33,821 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#192 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:33,838 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:33,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:33,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:33,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:33,852 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:33,852 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,852 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:33,852 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:33,853 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,853 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:33,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:33,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:33,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:33,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:33,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:33,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:33,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:33,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:33,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:33,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:33,864 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:33,864 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:33,864 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:54:33,865 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:33,865 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:33,865 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:33,865 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:33,865 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:33,865 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:34,251 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/ec3f156a3b6f42c8a625e52e8da559d9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/ec3f156a3b6f42c8a625e52e8da559d9 2023-04-25 09:54:34,256 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into ec3f156a3b6f42c8a625e52e8da559d9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:34,256 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,256 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=13, startTime=1682416473796; duration=0sec 2023-04-25 09:54:34,257 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:34,257 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:34,259 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:34,259 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:34,259 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,260 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5d10e10e9f5a43deaad3859b6c2e778e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/0037a6d972a84b178c09a78bbb195bb0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/368ef996a3714b0bb8d9ea52beb94b57] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=36.4 K 2023-04-25 09:54:34,260 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5d10e10e9f5a43deaad3859b6c2e778e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1682416469350 2023-04-25 09:54:34,260 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0037a6d972a84b178c09a78bbb195bb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1682416470229 2023-04-25 09:54:34,261 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 368ef996a3714b0bb8d9ea52beb94b57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416472385 2023-04-25 09:54:34,269 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#195 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:34,291 WARN [DataStreamer for file /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/7ce2a5837dd647aa9b42bf413fafb174] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-25 09:54:34,298 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042512ac178e865847099e061b949402c0e0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042512ac178e865847099e061b949402c0e0 2023-04-25 09:54:34,298 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/7ce2a5837dd647aa9b42bf413fafb174 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7ce2a5837dd647aa9b42bf413fafb174 2023-04-25 09:54:34,305 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 7ce2a5837dd647aa9b42bf413fafb174(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:34,305 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,306 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=13, startTime=1682416473796; duration=0sec 2023-04-25 09:54:34,306 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:34,322 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=516, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/2ea1f429336b476baa39dfc02e52965a 2023-04-25 09:54:34,348 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/218dededa2434863bb8fea1c768c4bdd 2023-04-25 09:54:34,363 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/a34187234734414299b7d08061c0a980 2023-04-25 09:54:34,369 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/2ea1f429336b476baa39dfc02e52965a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ea1f429336b476baa39dfc02e52965a 2023-04-25 09:54:34,374 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ea1f429336b476baa39dfc02e52965a, entries=100, sequenceid=516, filesize=18.6 K 2023-04-25 09:54:34,375 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/218dededa2434863bb8fea1c768c4bdd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/218dededa2434863bb8fea1c768c4bdd 2023-04-25 09:54:34,379 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/218dededa2434863bb8fea1c768c4bdd, entries=100, sequenceid=516, filesize=9.5 K 2023-04-25 09:54:34,380 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/a34187234734414299b7d08061c0a980 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a34187234734414299b7d08061c0a980 2023-04-25 09:54:34,384 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a34187234734414299b7d08061c0a980, entries=100, sequenceid=516, filesize=9.5 K 2023-04-25 09:54:34,385 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 5687fcb9918272e953b19ef4123710b6 in 521ms, sequenceid=516, compaction requested=false 2023-04-25 09:54:34,385 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,386 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:34,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:34,387 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,388 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:34,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c67c1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:34,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:34,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,397 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,397 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:34,398 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:34,398 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:34,398 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,398 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:34,399 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:34,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:34,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:34,402 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:34,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:34,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:34,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:34,402 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:34,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,407 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,409 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,409 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:34,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:34,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:34,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,410 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:34,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,410 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,510 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:34,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:34,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:34,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:34,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:34,515 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:34,515 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,515 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:34,515 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:34,515 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,515 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,518 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:34,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:34,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:34,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:34,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:34,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:34,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,530 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:34,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cea8f30[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:34,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:34,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,534 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:34,534 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,534 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:34,534 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:34,535 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:34,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:34,538 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:34,538 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:34,538 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:34,538 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:34,539 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:34,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,544 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:34,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:34,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,547 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:34,547 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:34,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,647 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:34,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:34,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:34,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:34,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:34,651 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:34,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:34,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:34,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,653 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:34,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:34,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:34,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:34,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:34,656 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,656 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:34,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:34,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,658 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,658 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,658 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:34,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a78e61b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:34,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:34,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:34,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:34,664 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,664 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:34,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:34,667 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:34,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:34,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:34,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:34,667 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:34,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,669 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,670 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:34,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:34,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,678 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,679 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:34,679 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:34,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,710 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/8774e8b81b5b4c0198cbc8520ef83c3b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8774e8b81b5b4c0198cbc8520ef83c3b 2023-04-25 09:54:34,716 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into 8774e8b81b5b4c0198cbc8520ef83c3b(size=26.2 K), total size for store is 44.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:34,716 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,716 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=13, startTime=1682416473796; duration=0sec 2023-04-25 09:54:34,716 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:34,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:34,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:34,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:34,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:34,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:34,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:34,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:34,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:34,762 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f0dd726b11cb4781bdeb0da7c2ac9090 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f0dd726b11cb4781bdeb0da7c2ac9090 2023-04-25 09:54:34,770 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=531, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/1fd3c2158d0e4ca19f02f73d4395a455 2023-04-25 09:54:34,779 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:34,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:34,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:34,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:34,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:34,784 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:34,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:34,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:34,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:34,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:34,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:34,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:34,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/846fe9a7256449eca53552a7fb22aa3a 2023-04-25 09:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:34,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:34,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:34,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:34,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:34,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:34,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:34,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,818 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/21df44ab9e1b4a79b67c33e016481aaa 2023-04-25 09:54:34,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:34,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f9e13c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:34,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:34,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:34,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:34,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/1fd3c2158d0e4ca19f02f73d4395a455 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1fd3c2158d0e4ca19f02f73d4395a455 2023-04-25 09:54:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:34,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416474807, max timestamp=1682416474824Num uniques 859; 2023-04-25 09:54:34,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416474807, max timestamp=1682416474824Num uniques 859; 2023-04-25 09:54:34,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416474807, max timestamp=1682416474824Num uniques 859; 2023-04-25 09:54:34,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:34,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:34,830 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:34,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:34,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:34,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:34,830 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:34,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1fd3c2158d0e4ca19f02f73d4395a455, entries=350, sequenceid=531, filesize=52.9 K 2023-04-25 09:54:34,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/846fe9a7256449eca53552a7fb22aa3a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/846fe9a7256449eca53552a7fb22aa3a 2023-04-25 09:54:34,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,835 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/846fe9a7256449eca53552a7fb22aa3a, entries=100, sequenceid=531, filesize=9.5 K 2023-04-25 09:54:34,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/21df44ab9e1b4a79b67c33e016481aaa as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/21df44ab9e1b4a79b67c33e016481aaa 2023-04-25 09:54:34,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:34,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:34,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:34,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,839 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:34,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,839 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/21df44ab9e1b4a79b67c33e016481aaa, entries=100, sequenceid=531, filesize=9.5 K 2023-04-25 09:54:34,843 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=60.38 KB/61830 for 5687fcb9918272e953b19ef4123710b6 in 112ms, sequenceid=531, compaction requested=true 2023-04-25 09:54:34,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:34,843 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:34,843 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:34,843 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:34,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:34,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=67.09 KB heapSize=181.08 KB 2023-04-25 09:54:34,845 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 100038 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:34,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:34,845 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:34,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:34,845 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:34,845 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32454 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:34,846 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:34,846 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:34,846 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8774e8b81b5b4c0198cbc8520ef83c3b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ea1f429336b476baa39dfc02e52965a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1fd3c2158d0e4ca19f02f73d4395a455] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=97.7 K 2023-04-25 09:54:34,846 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,846 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:34,846 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/ec3f156a3b6f42c8a625e52e8da559d9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/218dededa2434863bb8fea1c768c4bdd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/846fe9a7256449eca53552a7fb22aa3a] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=31.7 K 2023-04-25 09:54:34,846 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:34,846 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8774e8b81b5b4c0198cbc8520ef83c3b, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416472385 2023-04-25 09:54:34,846 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ec3f156a3b6f42c8a625e52e8da559d9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416472385 2023-04-25 09:54:34,847 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2ea1f429336b476baa39dfc02e52965a, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682416472545 2023-04-25 09:54:34,847 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 218dededa2434863bb8fea1c768c4bdd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682416472545 2023-04-25 09:54:34,847 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 846fe9a7256449eca53552a7fb22aa3a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682416474725 2023-04-25 09:54:34,848 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1fd3c2158d0e4ca19f02f73d4395a455, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682416474684 2023-04-25 09:54:34,858 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#201 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:34,881 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#202 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:34,908 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/2d17a424e0b341b68b2c9c63fb8c4b19 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2d17a424e0b341b68b2c9c63fb8c4b19 2023-04-25 09:54:34,913 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into 2d17a424e0b341b68b2c9c63fb8c4b19(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:34,913 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,913 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=13, startTime=1682416474843; duration=0sec 2023-04-25 09:54:34,913 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:34,913 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:34,915 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32454 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:34,915 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:34,915 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,915 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7ce2a5837dd647aa9b42bf413fafb174, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a34187234734414299b7d08061c0a980, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/21df44ab9e1b4a79b67c33e016481aaa] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=31.7 K 2023-04-25 09:54:34,916 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7ce2a5837dd647aa9b42bf413fafb174, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1682416472385 2023-04-25 09:54:34,916 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a34187234734414299b7d08061c0a980, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682416472545 2023-04-25 09:54:34,917 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 21df44ab9e1b4a79b67c33e016481aaa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682416474725 2023-04-25 09:54:34,924 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#204 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:34,930 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/92e9596e25054e2b87a7626e776bc510 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/92e9596e25054e2b87a7626e776bc510 2023-04-25 09:54:34,934 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into 92e9596e25054e2b87a7626e776bc510(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:34,934 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,935 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=13, startTime=1682416474843; duration=0sec 2023-04-25 09:54:34,935 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:34,939 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:34,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:34,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:34,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416534935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,948 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/8406e7e4df0b4a2c9e4919358f089486 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8406e7e4df0b4a2c9e4919358f089486 2023-04-25 09:54:34,953 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 8406e7e4df0b4a2c9e4919358f089486(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:34,953 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:34,953 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=13, startTime=1682416474843; duration=0sec 2023-04-25 09:54:34,953 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:34,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:34,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:34,956 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:34,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:34,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:34,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:34,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:34,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:34,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:34,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:34,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:34,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:34,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:34,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@210894cd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:34,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:34,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,984 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:34,984 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:34,986 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:34,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:34,989 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:34,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:34,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:34,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:34,990 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:34,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:34,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:34,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:34,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:34,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:34,999 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:34,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:34,999 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,000 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,000 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:35,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416535048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,100 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,104 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,107 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,107 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,107 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,111 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,111 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,112 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,112 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19231cd3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,117 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,117 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,120 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,120 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,123 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,123 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,126 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,131 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,131 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,231 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,235 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,236 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,236 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,240 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,252 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,252 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,252 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,252 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:35,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416535250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c669d2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,254 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,254 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,255 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,257 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,260 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,260 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,260 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,260 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,260 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,263 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,263 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,263 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,264 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,268 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,268 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,268 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,268 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,296 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258b3b1dc710d345e280a3b9dc1cda84e3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258b3b1dc710d345e280a3b9dc1cda84e3 2023-04-25 09:54:35,368 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,372 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,374 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,375 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,375 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,375 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,377 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,377 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,377 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,377 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,377 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,379 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,379 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,379 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,380 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39bb8e77[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,383 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,383 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,383 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,385 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,387 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,387 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,387 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,387 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,387 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,390 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,390 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,390 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,396 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,396 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,396 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,497 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,501 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,503 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,505 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,505 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,505 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,505 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,506 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,506 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,506 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,506 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,506 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,506 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,506 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,506 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,506 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,506 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,507 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,507 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,508 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf0db71[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,510 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,510 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,511 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,512 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,514 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,514 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,516 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,516 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,516 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,521 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,522 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,521 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,522 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:35,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416535555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,622 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,625 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,625 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,625 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,625 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,627 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,627 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,628 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,628 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,629 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,629 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,630 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,630 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,630 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,630 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d3481b8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,634 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,634 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,638 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,638 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,639 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,642 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,642 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,643 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,643 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,644 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,644 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,645 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,647 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,648 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,648 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,648 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,703 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=557, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/9b29e8149eee442bad20be8b5d12291b 2023-04-25 09:54:35,748 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,751 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,755 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,756 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,756 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,756 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,756 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,757 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a3beb35[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,759 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,759 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,760 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,760 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,761 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,761 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,762 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,764 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,765 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,768 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,768 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,768 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,772 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,772 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,773 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,773 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,873 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:35,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:35,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:35,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:35,876 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:35,876 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,876 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:35,876 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:35,876 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,878 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:35,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:35,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:35,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:35,880 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,880 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:35,880 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:35,880 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:35,881 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:35,883 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:35,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:35,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f161e33[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:35,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:35,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,887 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,887 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:35,887 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:35,887 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:35,887 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,887 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:35,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,889 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:35,889 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,889 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:35,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:35,891 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:35,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:35,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:35,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:35,891 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:35,893 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,893 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,893 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:35,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:35,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:35,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:35,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:35,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:35,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:35,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:35,897 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:35,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:35,897 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:35,998 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:35,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,001 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,005 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,005 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,006 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:36,006 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,007 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,007 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656771f3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,011 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,011 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,015 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,015 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,015 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,015 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,018 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,018 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,020 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,020 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,020 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,021 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,024 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,024 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,024 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:36,024 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:36,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416536062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/d22ee3908d1a4ab29af6f166bc2cc583 2023-04-25 09:54:36,124 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:36,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,128 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,129 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,130 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:36,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,134 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2715a6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,135 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,135 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,136 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,137 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,140 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,140 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,142 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:36,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,152 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,153 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,153 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:36,253 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:36,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,256 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,257 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,257 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,257 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,258 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,258 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,258 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,259 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:36,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,262 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,263 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c02754[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,264 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,264 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,265 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,265 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,265 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,268 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,268 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,270 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,270 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,270 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,270 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,270 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,272 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,272 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,272 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:36,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,276 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,276 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,276 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,276 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,376 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:36,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,379 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,379 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,381 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,381 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,383 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,383 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,383 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:36,384 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,387 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,387 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,387 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,387 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@162845d9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,388 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,388 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,390 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,390 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,391 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,391 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,391 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,391 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,394 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,394 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,395 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,395 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,395 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,397 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,404 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,404 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:36,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,504 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,508 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,509 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,516 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,516 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,518 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,518 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,518 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:36,518 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,520 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,521 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,521 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@481e4606[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,522 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,522 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,524 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/5b139c4fd818431fb528c11da0163790 2023-04-25 09:54:36,526 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,526 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,526 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,526 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/9b29e8149eee442bad20be8b5d12291b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9b29e8149eee442bad20be8b5d12291b 2023-04-25 09:54:36,530 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,530 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,532 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,532 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9b29e8149eee442bad20be8b5d12291b, entries=200, sequenceid=557, filesize=32.3 K 2023-04-25 09:54:36,535 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/d22ee3908d1a4ab29af6f166bc2cc583 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/d22ee3908d1a4ab29af6f166bc2cc583 2023-04-25 09:54:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,536 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,538 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/d22ee3908d1a4ab29af6f166bc2cc583, entries=150, sequenceid=557, filesize=11.9 K 2023-04-25 09:54:36,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/5b139c4fd818431fb528c11da0163790 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5b139c4fd818431fb528c11da0163790 2023-04-25 09:54:36,541 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,542 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5b139c4fd818431fb528c11da0163790, entries=150, sequenceid=557, filesize=11.9 K 2023-04-25 09:54:36,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,546 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,547 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~181.03 KB/185376, currentSize=114.05 KB/116790 for 5687fcb9918272e953b19ef4123710b6 in 1702ms, sequenceid=557, compaction requested=false 2023-04-25 09:54:36,547 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:36,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:36,647 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:36,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,650 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,652 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,654 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 09:54:36,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:36,654 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:36,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:36,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:36,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:36,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:36,673 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304259dd5c11559d44d3ea1f92f88a04ac4db to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259dd5c11559d44d3ea1f92f88a04ac4db 2023-04-25 09:54:36,689 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=580, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/24f59fcb04d145eaa53180e3ec032684 2023-04-25 09:54:36,716 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/8fc2b2ea2b514c558ab01f87453855b7 2023-04-25 09:54:36,742 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/cd4471bdb8f04a7cbc849d750feba377 2023-04-25 09:54:36,748 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/24f59fcb04d145eaa53180e3ec032684 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/24f59fcb04d145eaa53180e3ec032684 2023-04-25 09:54:36,752 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/24f59fcb04d145eaa53180e3ec032684, entries=150, sequenceid=580, filesize=25.4 K 2023-04-25 09:54:36,753 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/8fc2b2ea2b514c558ab01f87453855b7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/8fc2b2ea2b514c558ab01f87453855b7 2023-04-25 09:54:36,757 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/8fc2b2ea2b514c558ab01f87453855b7, entries=150, sequenceid=580, filesize=11.9 K 2023-04-25 09:54:36,757 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/cd4471bdb8f04a7cbc849d750feba377 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/cd4471bdb8f04a7cbc849d750feba377 2023-04-25 09:54:36,789 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/cd4471bdb8f04a7cbc849d750feba377, entries=150, sequenceid=580, filesize=11.9 K 2023-04-25 09:54:36,790 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 5687fcb9918272e953b19ef4123710b6 in 136ms, sequenceid=580, compaction requested=true 2023-04-25 09:54:36,790 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:36,790 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,795 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c30f086[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,799 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,799 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,799 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,803 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,804 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,808 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:36,812 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,812 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:36,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,913 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:36,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:36,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:36,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:36,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:36,917 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:36,917 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,917 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:36,917 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:36,917 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,917 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,919 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:36,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:36,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:36,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:36,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:36,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:36,932 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,932 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:36,932 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:36,932 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:36,934 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d280d3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:36,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:36,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,938 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:36,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:36,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:36,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:36,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:36,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:36,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:36,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:36,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:36,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:36,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:36,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:36,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:36,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:36,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:36,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:36,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:36,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:36,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:36,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:36,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:36,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:36,955 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:36,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:36,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:36,955 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:36,956 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:37,056 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:37,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:37,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:37,060 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:37,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:37,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:37,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:37,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,062 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:37,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:37,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:37,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:37,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:37,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:37,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:37,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:37,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:37,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2080051d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:37,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:37,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,070 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,070 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:37,070 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:37,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:37,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:37,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:37,075 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:37,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:37,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:37,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:37,075 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:37,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:37,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:37,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:37,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,084 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:37,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:37,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:37,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:37,084 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:37,084 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:37,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:37,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:54:37,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:37,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:37,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:37,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:37,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:37,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:37,184 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:37,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:37,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:37,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:37,200 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:37,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:37,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:37,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:37,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:37,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:37,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:37,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:37,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:37,209 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:37,209 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:37,209 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:37,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:37,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:37,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:37,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:37,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:37,210 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f8dd7944206943c0bfc2f1d9d33a3b43 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f8dd7944206943c0bfc2f1d9d33a3b43 2023-04-25 09:54:37,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:37,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:37,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:37,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a94597d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:37,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:37,218 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,219 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:37,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:37,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:37,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:37,221 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:37,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:37,225 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:37,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:37,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:37,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:37,225 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:37,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:37,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:37,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,230 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,232 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=592, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/87467366370744d0ab75692b9e4f4523 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:37,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:37,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:37,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:37,235 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:37,235 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:37,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:37,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416477220, max timestamp=1682416477260Num uniques 859; 2023-04-25 09:54:37,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416477220, max timestamp=1682416477260Num uniques 859; 2023-04-25 09:54:37,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:37,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416477220, max timestamp=1682416477260Num uniques 859; 2023-04-25 09:54:37,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:54:37,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416537328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,335 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:37,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:37,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:37,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:37,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:37,339 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:37,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:37,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:37,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:37,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,341 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:37,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:37,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:37,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:37,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:37,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:37,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:37,343 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:37,343 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. started... 2023-04-25 09:54:37,343 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. as already flushing 2023-04-25 09:54:37,344 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:37,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,346 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:37,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71cb2e69[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:37,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:37,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,350 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:37,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:37,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:37,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:37,352 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:37,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:37,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:37,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:37,353 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:37,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:37,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:37,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,357 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:37,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,361 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:37,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:37,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:37,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:37,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:37,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:37,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:37,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:37,361 DEBUG [Thread-926] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:37,361 DEBUG [Thread-926] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:37,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:37,432 DEBUG [Thread-927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:60014 2023-04-25 09:54:37,432 DEBUG [Thread-927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:37,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416537431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,433 DEBUG [Thread-929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:60014 2023-04-25 09:54:37,433 DEBUG [Thread-929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:37,436 DEBUG [Thread-931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:60014 2023-04-25 09:54:37,436 DEBUG [Thread-931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:37,440 DEBUG [Thread-933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58e27a32 to 127.0.0.1:60014 2023-04-25 09:54:37,441 DEBUG [Thread-933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:37,462 DEBUG [Thread-926] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:37,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:37,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44106 deadline: 1682416537499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,501 DEBUG [Thread-924] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:37,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44090 deadline: 1682416537519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,521 DEBUG [Thread-916] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:37,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44076 deadline: 1682416537521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,523 DEBUG [Thread-920] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:37,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44144 deadline: 1682416537523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,524 DEBUG [Thread-918] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:37,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416537636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:37,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/6883ce6ccb0747509bffa2ca8c46ce54 2023-04-25 09:54:37,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:37,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44112 deadline: 1682416537939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:38,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/b239958f2eaa4a6ba2f5a8a98a92ddd6 2023-04-25 09:54:38,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/87467366370744d0ab75692b9e4f4523 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/87467366370744d0ab75692b9e4f4523 2023-04-25 09:54:38,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/87467366370744d0ab75692b9e4f4523, entries=150, sequenceid=592, filesize=25.4 K 2023-04-25 09:54:38,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/6883ce6ccb0747509bffa2ca8c46ce54 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6883ce6ccb0747509bffa2ca8c46ce54 2023-04-25 09:54:38,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6883ce6ccb0747509bffa2ca8c46ce54, entries=150, sequenceid=592, filesize=11.9 K 2023-04-25 09:54:38,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/b239958f2eaa4a6ba2f5a8a98a92ddd6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b239958f2eaa4a6ba2f5a8a98a92ddd6 2023-04-25 09:54:38,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b239958f2eaa4a6ba2f5a8a98a92ddd6, entries=150, sequenceid=592, filesize=11.9 K 2023-04-25 09:54:38,077 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5687fcb9918272e953b19ef4123710b6 in 900ms, sequenceid=592, compaction requested=true 2023-04-25 09:54:38,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:38,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:38,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:38,077 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:38,077 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:38,077 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:38,078 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112110 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:38,078 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:38,078 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/A is initiating minor compaction (all files) 2023-04-25 09:54:38,078 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/B is initiating minor compaction (all files) 2023-04-25 09:54:38,078 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/A in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:38,078 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/B in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:38,078 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/92e9596e25054e2b87a7626e776bc510, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9b29e8149eee442bad20be8b5d12291b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/24f59fcb04d145eaa53180e3ec032684, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/87467366370744d0ab75692b9e4f4523] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=109.5 K 2023-04-25 09:54:38,078 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2d17a424e0b341b68b2c9c63fb8c4b19, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/d22ee3908d1a4ab29af6f166bc2cc583, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/8fc2b2ea2b514c558ab01f87453855b7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6883ce6ccb0747509bffa2ca8c46ce54] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=48.5 K 2023-04-25 09:54:38,079 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 92e9596e25054e2b87a7626e776bc510, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682416472507 2023-04-25 09:54:38,079 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2d17a424e0b341b68b2c9c63fb8c4b19, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682416472507 2023-04-25 09:54:38,079 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9b29e8149eee442bad20be8b5d12291b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1682416474816 2023-04-25 09:54:38,079 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d22ee3908d1a4ab29af6f166bc2cc583, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1682416474830 2023-04-25 09:54:38,079 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 24f59fcb04d145eaa53180e3ec032684, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1682416474907 2023-04-25 09:54:38,080 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8fc2b2ea2b514c558ab01f87453855b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1682416474907 2023-04-25 09:54:38,080 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 87467366370744d0ab75692b9e4f4523, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1682416477169 2023-04-25 09:54:38,080 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6883ce6ccb0747509bffa2ca8c46ce54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1682416477169 2023-04-25 09:54:38,088 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#B#compaction#214 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:38,097 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#A#compaction#213 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:38,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:38,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-25 09:54:38,445 DEBUG [Thread-922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:60014 2023-04-25 09:54:38,445 DEBUG [Thread-922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:38,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:38,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:38,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:38,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:38,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:38,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:38,502 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/b1923cec5ba74aa3ba303324d57a7a10 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b1923cec5ba74aa3ba303324d57a7a10 2023-04-25 09:54:38,507 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/B of 5687fcb9918272e953b19ef4123710b6 into b1923cec5ba74aa3ba303324d57a7a10(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:38,507 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:38,507 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/B, priority=12, startTime=1682416478077; duration=0sec 2023-04-25 09:54:38,507 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:38,507 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:38,508 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:38,508 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 5687fcb9918272e953b19ef4123710b6/C is initiating minor compaction (all files) 2023-04-25 09:54:38,508 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5687fcb9918272e953b19ef4123710b6/C in TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:38,508 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8406e7e4df0b4a2c9e4919358f089486, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5b139c4fd818431fb528c11da0163790, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/cd4471bdb8f04a7cbc849d750feba377, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b239958f2eaa4a6ba2f5a8a98a92ddd6] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp, totalSize=48.5 K 2023-04-25 09:54:38,508 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8406e7e4df0b4a2c9e4919358f089486, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1682416472507 2023-04-25 09:54:38,509 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5b139c4fd818431fb528c11da0163790, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1682416474830 2023-04-25 09:54:38,509 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting cd4471bdb8f04a7cbc849d750feba377, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1682416474907 2023-04-25 09:54:38,509 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b239958f2eaa4a6ba2f5a8a98a92ddd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1682416477169 2023-04-25 09:54:38,515 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 5687fcb9918272e953b19ef4123710b6#C#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-25 09:54:38,860 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254b0f7ffb2e2c46c8810dd3e769c87db5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254b0f7ffb2e2c46c8810dd3e769c87db5 2023-04-25 09:54:38,907 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.07692307692307693, tune throughput to 53.85 MB/second 2023-04-25 09:54:38,923 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/31b6860013494bf1a7cd0f3a571d8bca as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/31b6860013494bf1a7cd0f3a571d8bca 2023-04-25 09:54:38,927 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/C of 5687fcb9918272e953b19ef4123710b6 into 31b6860013494bf1a7cd0f3a571d8bca(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:38,927 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:38,927 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/C, priority=12, startTime=1682416478077; duration=0sec 2023-04-25 09:54:38,927 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:39,265 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=629, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/f7fe219390b241dda2fd23a2966a6768 2023-04-25 09:54:39,315 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/b5fa6e419f5e441e92dfd00c9eafab29 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/b5fa6e419f5e441e92dfd00c9eafab29 2023-04-25 09:54:39,319 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5687fcb9918272e953b19ef4123710b6/A of 5687fcb9918272e953b19ef4123710b6 into b5fa6e419f5e441e92dfd00c9eafab29(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 09:54:39,319 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:39,319 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6., storeName=5687fcb9918272e953b19ef4123710b6/A, priority=12, startTime=1682416478077; duration=1sec 2023-04-25 09:54:39,319 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:39,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/4ac596561dce49c5a70b0bf04e693cbf 2023-04-25 09:54:40,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/e458d098545a4fffa3d1dd6c0b5e9771 2023-04-25 09:54:40,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/f7fe219390b241dda2fd23a2966a6768 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f7fe219390b241dda2fd23a2966a6768 2023-04-25 09:54:40,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f7fe219390b241dda2fd23a2966a6768, entries=150, sequenceid=629, filesize=25.4 K 2023-04-25 09:54:40,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/4ac596561dce49c5a70b0bf04e693cbf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/4ac596561dce49c5a70b0bf04e693cbf 2023-04-25 09:54:40,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/4ac596561dce49c5a70b0bf04e693cbf, entries=150, sequenceid=629, filesize=11.9 K 2023-04-25 09:54:40,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/e458d098545a4fffa3d1dd6c0b5e9771 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/e458d098545a4fffa3d1dd6c0b5e9771 2023-04-25 09:54:40,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/e458d098545a4fffa3d1dd6c0b5e9771, entries=150, sequenceid=629, filesize=11.9 K 2023-04-25 09:54:40,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for 5687fcb9918272e953b19ef4123710b6 in 1658ms, sequenceid=629, compaction requested=false 2023-04-25 09:54:40,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:47,529 DEBUG [Thread-920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:60014 2023-04-25 09:54:47,529 DEBUG [Thread-920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:47,530 DEBUG [Thread-916] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x051c62b6 to 127.0.0.1:60014 2023-04-25 09:54:47,530 DEBUG [Thread-916] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:47,579 DEBUG [Thread-924] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:60014 2023-04-25 09:54:47,580 DEBUG [Thread-924] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:47,620 DEBUG [Thread-918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:60014 2023-04-25 09:54:47,621 DEBUG [Thread-918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 167 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 5250 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 5251 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 2202 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 6604 rows 2023-04-25 09:54:47,621 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 2210 2023-04-25 09:54:47,622 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 6629 rows 2023-04-25 09:54:47,622 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:54:47,622 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b8d0eb2 to 127.0.0.1:60014 2023-04-25 09:54:47,622 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:54:47,623 INFO [Listener at localhost/33705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 09:54:47,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 09:54:47,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 09:54:47,630 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416487630"}]},"ts":"1682416487630"} 2023-04-25 09:54:47,631 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 09:54:47,632 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 09:54:47,633 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, UNASSIGN}] 2023-04-25 09:54:47,634 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, UNASSIGN 2023-04-25 09:54:47,635 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:47,635 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416487635"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416487635"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416487635"}]},"ts":"1682416487635"} 2023-04-25 09:54:47,636 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:54:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 09:54:47,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5687fcb9918272e953b19ef4123710b6, disabling compactions & flushes 2023-04-25 09:54:47,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. after waiting 0 ms 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:47,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5687fcb9918272e953b19ef4123710b6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=A 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=B 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5687fcb9918272e953b19ef4123710b6, store=C 2023-04-25 09:54:47,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:47,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 09:54:48,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ebf6b41b26e6449ca63ca83eba35e802 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ebf6b41b26e6449ca63ca83eba35e802 2023-04-25 09:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 09:54:48,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=639, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/1d6e4bf1c4894202ae0a0872c1ad3834 2023-04-25 09:54:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 09:54:49,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/65eb6ee2dd364516840903b8b3f2b71e 2023-04-25 09:54:49,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/4b549ae0e6574c7aa3bfb49d3bdb4997 2023-04-25 09:54:49,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/A/1d6e4bf1c4894202ae0a0872c1ad3834 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1d6e4bf1c4894202ae0a0872c1ad3834 2023-04-25 09:54:49,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1d6e4bf1c4894202ae0a0872c1ad3834, entries=100, sequenceid=639, filesize=18.6 K 2023-04-25 09:54:49,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/B/65eb6ee2dd364516840903b8b3f2b71e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/65eb6ee2dd364516840903b8b3f2b71e 2023-04-25 09:54:49,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/65eb6ee2dd364516840903b8b3f2b71e, entries=100, sequenceid=639, filesize=9.5 K 2023-04-25 09:54:49,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/.tmp/C/4b549ae0e6574c7aa3bfb49d3bdb4997 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/4b549ae0e6574c7aa3bfb49d3bdb4997 2023-04-25 09:54:49,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/4b549ae0e6574c7aa3bfb49d3bdb4997, entries=100, sequenceid=639, filesize=9.5 K 2023-04-25 09:54:49,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5687fcb9918272e953b19ef4123710b6 in 1652ms, sequenceid=639, compaction requested=true 2023-04-25 09:54:49,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f3b49d54998442109692a176a7ca91b8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/3a2b24ed1db74685a8b1fd54bd3a8a11, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9c74fb5a3af843d89334cec6246139a2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/c02aca29e13e4d73b67b04a20332a197, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/480f3b56f2b94a27b0493d4fa264bc2c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/4ae2940d9b2748ca909628bffcdd0410, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/683ff37db2874d119d4335d18e0e393b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/6b0fdfe64a86432290cdc115d3ad0aea, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/505eba8f6ae143658468a02daee4c4af, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/5352c271c2af4be0b751bda610cc661f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/71ca5fae67b24affa806382ee0f1b2c5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8678d733cb2f452fad64a2a71137f4bb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ab59f0516af440f96960235c45ce954, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e6a6d3179ae746da8ea7590d688d2d0b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/acc32421ca764d65bb8039ef465f8bb4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/11dbc147aaa54c7597b9fa90abcc78a0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/ab90d9dd5bac4796acbe8424c3eadcf6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/d7d3c4d76fc84a7cbcdc0502507d4ea0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2bb4d89a8c5a4c689c9e6507b412ede4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/94e0f75211f04baab3316835ebd5a44f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e02d956d6cac4cdf8b1306cfc36f57be, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/fc82061188d344bdb45a8f6cddba000e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8ec67f2c295946e39f0ad3367d9b58a3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8774e8b81b5b4c0198cbc8520ef83c3b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ea1f429336b476baa39dfc02e52965a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1fd3c2158d0e4ca19f02f73d4395a455, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/92e9596e25054e2b87a7626e776bc510, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9b29e8149eee442bad20be8b5d12291b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/24f59fcb04d145eaa53180e3ec032684, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/87467366370744d0ab75692b9e4f4523] to archive 2023-04-25 09:54:49,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:54:49,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f3b49d54998442109692a176a7ca91b8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f3b49d54998442109692a176a7ca91b8 2023-04-25 09:54:49,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/3a2b24ed1db74685a8b1fd54bd3a8a11 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/3a2b24ed1db74685a8b1fd54bd3a8a11 2023-04-25 09:54:49,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9c74fb5a3af843d89334cec6246139a2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9c74fb5a3af843d89334cec6246139a2 2023-04-25 09:54:49,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/c02aca29e13e4d73b67b04a20332a197 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/c02aca29e13e4d73b67b04a20332a197 2023-04-25 09:54:49,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/480f3b56f2b94a27b0493d4fa264bc2c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/480f3b56f2b94a27b0493d4fa264bc2c 2023-04-25 09:54:49,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/4ae2940d9b2748ca909628bffcdd0410 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/4ae2940d9b2748ca909628bffcdd0410 2023-04-25 09:54:49,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/683ff37db2874d119d4335d18e0e393b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/683ff37db2874d119d4335d18e0e393b 2023-04-25 09:54:49,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/6b0fdfe64a86432290cdc115d3ad0aea to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/6b0fdfe64a86432290cdc115d3ad0aea 2023-04-25 09:54:49,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/505eba8f6ae143658468a02daee4c4af to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/505eba8f6ae143658468a02daee4c4af 2023-04-25 09:54:49,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/5352c271c2af4be0b751bda610cc661f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/5352c271c2af4be0b751bda610cc661f 2023-04-25 09:54:49,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/71ca5fae67b24affa806382ee0f1b2c5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/71ca5fae67b24affa806382ee0f1b2c5 2023-04-25 09:54:49,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8678d733cb2f452fad64a2a71137f4bb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8678d733cb2f452fad64a2a71137f4bb 2023-04-25 09:54:49,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ab59f0516af440f96960235c45ce954 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ab59f0516af440f96960235c45ce954 2023-04-25 09:54:49,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e6a6d3179ae746da8ea7590d688d2d0b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e6a6d3179ae746da8ea7590d688d2d0b 2023-04-25 09:54:49,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/acc32421ca764d65bb8039ef465f8bb4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/acc32421ca764d65bb8039ef465f8bb4 2023-04-25 09:54:49,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/11dbc147aaa54c7597b9fa90abcc78a0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/11dbc147aaa54c7597b9fa90abcc78a0 2023-04-25 09:54:49,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/ab90d9dd5bac4796acbe8424c3eadcf6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/ab90d9dd5bac4796acbe8424c3eadcf6 2023-04-25 09:54:49,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/d7d3c4d76fc84a7cbcdc0502507d4ea0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/d7d3c4d76fc84a7cbcdc0502507d4ea0 2023-04-25 09:54:49,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2bb4d89a8c5a4c689c9e6507b412ede4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2bb4d89a8c5a4c689c9e6507b412ede4 2023-04-25 09:54:49,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/94e0f75211f04baab3316835ebd5a44f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/94e0f75211f04baab3316835ebd5a44f 2023-04-25 09:54:49,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e02d956d6cac4cdf8b1306cfc36f57be to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/e02d956d6cac4cdf8b1306cfc36f57be 2023-04-25 09:54:49,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/fc82061188d344bdb45a8f6cddba000e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/fc82061188d344bdb45a8f6cddba000e 2023-04-25 09:54:49,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8ec67f2c295946e39f0ad3367d9b58a3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8ec67f2c295946e39f0ad3367d9b58a3 2023-04-25 09:54:49,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8774e8b81b5b4c0198cbc8520ef83c3b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/8774e8b81b5b4c0198cbc8520ef83c3b 2023-04-25 09:54:49,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ea1f429336b476baa39dfc02e52965a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/2ea1f429336b476baa39dfc02e52965a 2023-04-25 09:54:49,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1fd3c2158d0e4ca19f02f73d4395a455 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1fd3c2158d0e4ca19f02f73d4395a455 2023-04-25 09:54:49,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/92e9596e25054e2b87a7626e776bc510 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/92e9596e25054e2b87a7626e776bc510 2023-04-25 09:54:49,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9b29e8149eee442bad20be8b5d12291b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/9b29e8149eee442bad20be8b5d12291b 2023-04-25 09:54:49,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/24f59fcb04d145eaa53180e3ec032684 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/24f59fcb04d145eaa53180e3ec032684 2023-04-25 09:54:49,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/87467366370744d0ab75692b9e4f4523 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/87467366370744d0ab75692b9e4f4523 2023-04-25 09:54:49,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/dcff060b05744e159e230896ff283124, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/5c39df869d1045309736e97c5dc3ccc4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/17ebfe6943ae458bbddea87750f39dfc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b22bcff7f1924195b6bce80aa4d4ce79, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/837985e72e1c486f9175721fe63634a1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/e9611b555890443abeeb4d36b580fcab, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6fb9c16345f5471dabe65bc48d0e9dfb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1833e4c727dd432f9004c51940e8bcc8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2035c8ecf3be4f6d8e50bc78da22ad4b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/7d33c142fab84810a9d2e401bc10626e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/9c30d67de41c41919ba68e0f4d938b75, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/c73b79ad22f9445397df291b8154768f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/a2dd003cc8174d67bed3251968258e1c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/aed15c2932a540b18a8cc55166de8245, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1504af33a12f47f0828b00a46fb5e684, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f4d89abd5e3049869816172582f11006, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f71ac8ded8504c5c893fa831237cf905, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/3330ad00429e4fa98fc134a3e23b2e87, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/660d03f186004556bc875c67abbce58b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/335538b44466463682fa75969601bafa, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/71c6f029c5f84cd69eb075160ded1874, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/07b83fa464434c37b3d811141a49243e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/ec3f156a3b6f42c8a625e52e8da559d9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/67d395aa9c29475bbd75a25cc6b89566, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/218dededa2434863bb8fea1c768c4bdd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2d17a424e0b341b68b2c9c63fb8c4b19, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/846fe9a7256449eca53552a7fb22aa3a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/d22ee3908d1a4ab29af6f166bc2cc583, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/8fc2b2ea2b514c558ab01f87453855b7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6883ce6ccb0747509bffa2ca8c46ce54] to archive 2023-04-25 09:54:49,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:54:49,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/dcff060b05744e159e230896ff283124 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/dcff060b05744e159e230896ff283124 2023-04-25 09:54:49,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/5c39df869d1045309736e97c5dc3ccc4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/5c39df869d1045309736e97c5dc3ccc4 2023-04-25 09:54:49,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/17ebfe6943ae458bbddea87750f39dfc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/17ebfe6943ae458bbddea87750f39dfc 2023-04-25 09:54:49,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b22bcff7f1924195b6bce80aa4d4ce79 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b22bcff7f1924195b6bce80aa4d4ce79 2023-04-25 09:54:49,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/837985e72e1c486f9175721fe63634a1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/837985e72e1c486f9175721fe63634a1 2023-04-25 09:54:49,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/e9611b555890443abeeb4d36b580fcab to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/e9611b555890443abeeb4d36b580fcab 2023-04-25 09:54:49,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6fb9c16345f5471dabe65bc48d0e9dfb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6fb9c16345f5471dabe65bc48d0e9dfb 2023-04-25 09:54:49,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1833e4c727dd432f9004c51940e8bcc8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1833e4c727dd432f9004c51940e8bcc8 2023-04-25 09:54:49,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2035c8ecf3be4f6d8e50bc78da22ad4b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2035c8ecf3be4f6d8e50bc78da22ad4b 2023-04-25 09:54:49,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/7d33c142fab84810a9d2e401bc10626e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/7d33c142fab84810a9d2e401bc10626e 2023-04-25 09:54:49,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/9c30d67de41c41919ba68e0f4d938b75 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/9c30d67de41c41919ba68e0f4d938b75 2023-04-25 09:54:49,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/c73b79ad22f9445397df291b8154768f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/c73b79ad22f9445397df291b8154768f 2023-04-25 09:54:49,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/a2dd003cc8174d67bed3251968258e1c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/a2dd003cc8174d67bed3251968258e1c 2023-04-25 09:54:49,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/aed15c2932a540b18a8cc55166de8245 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/aed15c2932a540b18a8cc55166de8245 2023-04-25 09:54:49,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1504af33a12f47f0828b00a46fb5e684 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/1504af33a12f47f0828b00a46fb5e684 2023-04-25 09:54:49,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f4d89abd5e3049869816172582f11006 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f4d89abd5e3049869816172582f11006 2023-04-25 09:54:49,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f71ac8ded8504c5c893fa831237cf905 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/f71ac8ded8504c5c893fa831237cf905 2023-04-25 09:54:49,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/3330ad00429e4fa98fc134a3e23b2e87 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/3330ad00429e4fa98fc134a3e23b2e87 2023-04-25 09:54:49,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/660d03f186004556bc875c67abbce58b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/660d03f186004556bc875c67abbce58b 2023-04-25 09:54:49,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/335538b44466463682fa75969601bafa to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/335538b44466463682fa75969601bafa 2023-04-25 09:54:49,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/71c6f029c5f84cd69eb075160ded1874 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/71c6f029c5f84cd69eb075160ded1874 2023-04-25 09:54:49,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/07b83fa464434c37b3d811141a49243e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/07b83fa464434c37b3d811141a49243e 2023-04-25 09:54:49,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/ec3f156a3b6f42c8a625e52e8da559d9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/ec3f156a3b6f42c8a625e52e8da559d9 2023-04-25 09:54:49,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/67d395aa9c29475bbd75a25cc6b89566 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/67d395aa9c29475bbd75a25cc6b89566 2023-04-25 09:54:49,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/218dededa2434863bb8fea1c768c4bdd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/218dededa2434863bb8fea1c768c4bdd 2023-04-25 09:54:49,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2d17a424e0b341b68b2c9c63fb8c4b19 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/2d17a424e0b341b68b2c9c63fb8c4b19 2023-04-25 09:54:49,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/846fe9a7256449eca53552a7fb22aa3a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/846fe9a7256449eca53552a7fb22aa3a 2023-04-25 09:54:49,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/d22ee3908d1a4ab29af6f166bc2cc583 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/d22ee3908d1a4ab29af6f166bc2cc583 2023-04-25 09:54:49,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/8fc2b2ea2b514c558ab01f87453855b7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/8fc2b2ea2b514c558ab01f87453855b7 2023-04-25 09:54:49,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6883ce6ccb0747509bffa2ca8c46ce54 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/6883ce6ccb0747509bffa2ca8c46ce54 2023-04-25 09:54:49,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c90619f3507745d9af68ce1e6a8d4844, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b9c3be2174ac4eb0a1d4a12dba150fa0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7cd9008a628540d980753e22ead06b0c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/54c499fecd2f44edb5a438f2d491dbff, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c4810d9a561549708d3b3cb2ff79a4e1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8451102c859f433991c3ec4f347ed22a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/ca28cce4e0da4a8fa85d49f71392aff6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8c8fa2c315f640fa98ca67e6fc80da0a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/6c50f56c426d45329c54be0d405a9660, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c101d5933bd74117b764384554201e94, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/08ea4a6c84c84c3d8bd11c08cd70ef72, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a9cbf5c8089e4392b541f495f07ef613, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/942b160c59264e3e930e19aebf09a8b4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/437684168133401884c9587a1340437b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/57bae6260c11449f9d30b141ef961bed, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a746622f23db4b40b58a9baeafdd028c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c37d345fd4e54d65aa8159765d696633, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/00858272fb754769931624ef76214306, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/f94d2fadc6824dadb92923d485dcee57, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5d10e10e9f5a43deaad3859b6c2e778e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/afb3facb4bac4d47b624225e6f1e260e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/0037a6d972a84b178c09a78bbb195bb0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7ce2a5837dd647aa9b42bf413fafb174, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/368ef996a3714b0bb8d9ea52beb94b57, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a34187234734414299b7d08061c0a980, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8406e7e4df0b4a2c9e4919358f089486, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/21df44ab9e1b4a79b67c33e016481aaa, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5b139c4fd818431fb528c11da0163790, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/cd4471bdb8f04a7cbc849d750feba377, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b239958f2eaa4a6ba2f5a8a98a92ddd6] to archive 2023-04-25 09:54:49,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:54:49,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c90619f3507745d9af68ce1e6a8d4844 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c90619f3507745d9af68ce1e6a8d4844 2023-04-25 09:54:49,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b9c3be2174ac4eb0a1d4a12dba150fa0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b9c3be2174ac4eb0a1d4a12dba150fa0 2023-04-25 09:54:49,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7cd9008a628540d980753e22ead06b0c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7cd9008a628540d980753e22ead06b0c 2023-04-25 09:54:49,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/54c499fecd2f44edb5a438f2d491dbff to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/54c499fecd2f44edb5a438f2d491dbff 2023-04-25 09:54:49,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c4810d9a561549708d3b3cb2ff79a4e1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c4810d9a561549708d3b3cb2ff79a4e1 2023-04-25 09:54:49,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8451102c859f433991c3ec4f347ed22a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8451102c859f433991c3ec4f347ed22a 2023-04-25 09:54:49,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/ca28cce4e0da4a8fa85d49f71392aff6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/ca28cce4e0da4a8fa85d49f71392aff6 2023-04-25 09:54:49,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8c8fa2c315f640fa98ca67e6fc80da0a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8c8fa2c315f640fa98ca67e6fc80da0a 2023-04-25 09:54:49,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/6c50f56c426d45329c54be0d405a9660 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/6c50f56c426d45329c54be0d405a9660 2023-04-25 09:54:49,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c101d5933bd74117b764384554201e94 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c101d5933bd74117b764384554201e94 2023-04-25 09:54:49,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/08ea4a6c84c84c3d8bd11c08cd70ef72 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/08ea4a6c84c84c3d8bd11c08cd70ef72 2023-04-25 09:54:49,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a9cbf5c8089e4392b541f495f07ef613 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a9cbf5c8089e4392b541f495f07ef613 2023-04-25 09:54:49,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/942b160c59264e3e930e19aebf09a8b4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/942b160c59264e3e930e19aebf09a8b4 2023-04-25 09:54:49,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/437684168133401884c9587a1340437b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/437684168133401884c9587a1340437b 2023-04-25 09:54:49,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/57bae6260c11449f9d30b141ef961bed to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/57bae6260c11449f9d30b141ef961bed 2023-04-25 09:54:49,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a746622f23db4b40b58a9baeafdd028c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a746622f23db4b40b58a9baeafdd028c 2023-04-25 09:54:49,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c37d345fd4e54d65aa8159765d696633 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/c37d345fd4e54d65aa8159765d696633 2023-04-25 09:54:49,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/00858272fb754769931624ef76214306 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/00858272fb754769931624ef76214306 2023-04-25 09:54:49,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/f94d2fadc6824dadb92923d485dcee57 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/f94d2fadc6824dadb92923d485dcee57 2023-04-25 09:54:49,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5d10e10e9f5a43deaad3859b6c2e778e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5d10e10e9f5a43deaad3859b6c2e778e 2023-04-25 09:54:49,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/afb3facb4bac4d47b624225e6f1e260e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/afb3facb4bac4d47b624225e6f1e260e 2023-04-25 09:54:49,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/0037a6d972a84b178c09a78bbb195bb0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/0037a6d972a84b178c09a78bbb195bb0 2023-04-25 09:54:49,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7ce2a5837dd647aa9b42bf413fafb174 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/7ce2a5837dd647aa9b42bf413fafb174 2023-04-25 09:54:49,547 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/368ef996a3714b0bb8d9ea52beb94b57 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/368ef996a3714b0bb8d9ea52beb94b57 2023-04-25 09:54:49,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a34187234734414299b7d08061c0a980 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/a34187234734414299b7d08061c0a980 2023-04-25 09:54:49,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8406e7e4df0b4a2c9e4919358f089486 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/8406e7e4df0b4a2c9e4919358f089486 2023-04-25 09:54:49,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/21df44ab9e1b4a79b67c33e016481aaa to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/21df44ab9e1b4a79b67c33e016481aaa 2023-04-25 09:54:49,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5b139c4fd818431fb528c11da0163790 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/5b139c4fd818431fb528c11da0163790 2023-04-25 09:54:49,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/cd4471bdb8f04a7cbc849d750feba377 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/cd4471bdb8f04a7cbc849d750feba377 2023-04-25 09:54:49,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b239958f2eaa4a6ba2f5a8a98a92ddd6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/b239958f2eaa4a6ba2f5a8a98a92ddd6 2023-04-25 09:54:49,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/recovered.edits/642.seqid, newMaxSeqId=642, maxSeqId=4 2023-04-25 09:54:49,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6. 2023-04-25 09:54:49,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5687fcb9918272e953b19ef4123710b6: 2023-04-25 09:54:49,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:49,559 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=5687fcb9918272e953b19ef4123710b6, regionState=CLOSED 2023-04-25 09:54:49,559 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416489559"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416489559"}]},"ts":"1682416489559"} 2023-04-25 09:54:49,561 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-25 09:54:49,561 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 5687fcb9918272e953b19ef4123710b6, server=jenkins-hbase4.apache.org,39407,1682416417123 in 1.9240 sec 2023-04-25 09:54:49,563 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-25 09:54:49,563 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5687fcb9918272e953b19ef4123710b6, UNASSIGN in 1.9280 sec 2023-04-25 09:54:49,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416489563"}]},"ts":"1682416489563"} 2023-04-25 09:54:49,564 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 09:54:49,566 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 09:54:49,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9420 sec 2023-04-25 09:54:49,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-25 09:54:49,734 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-25 09:54:49,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 09:54:49,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,737 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-25 09:54:49,738 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,743 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:49,745 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/recovered.edits] 2023-04-25 09:54:49,748 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1d6e4bf1c4894202ae0a0872c1ad3834 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/1d6e4bf1c4894202ae0a0872c1ad3834 2023-04-25 09:54:49,749 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/b5fa6e419f5e441e92dfd00c9eafab29 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/b5fa6e419f5e441e92dfd00c9eafab29 2023-04-25 09:54:49,750 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f7fe219390b241dda2fd23a2966a6768 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/A/f7fe219390b241dda2fd23a2966a6768 2023-04-25 09:54:49,753 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/4ac596561dce49c5a70b0bf04e693cbf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/4ac596561dce49c5a70b0bf04e693cbf 2023-04-25 09:54:49,754 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/65eb6ee2dd364516840903b8b3f2b71e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/65eb6ee2dd364516840903b8b3f2b71e 2023-04-25 09:54:49,755 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b1923cec5ba74aa3ba303324d57a7a10 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/B/b1923cec5ba74aa3ba303324d57a7a10 2023-04-25 09:54:49,757 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/31b6860013494bf1a7cd0f3a571d8bca to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/31b6860013494bf1a7cd0f3a571d8bca 2023-04-25 09:54:49,758 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/4b549ae0e6574c7aa3bfb49d3bdb4997 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/4b549ae0e6574c7aa3bfb49d3bdb4997 2023-04-25 09:54:49,759 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/e458d098545a4fffa3d1dd6c0b5e9771 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/C/e458d098545a4fffa3d1dd6c0b5e9771 2023-04-25 09:54:49,761 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/recovered.edits/642.seqid to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6/recovered.edits/642.seqid 2023-04-25 09:54:49,761 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/5687fcb9918272e953b19ef4123710b6 2023-04-25 09:54:49,762 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:54:49,762 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 09:54:49,763 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 09:54:49,766 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425004ce414fbb74414ba56c218a746d0e2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425004ce414fbb74414ba56c218a746d0e2 2023-04-25 09:54:49,767 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250a49181156124e10b87cac263d24d204 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304250a49181156124e10b87cac263d24d204 2023-04-25 09:54:49,768 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042512ac178e865847099e061b949402c0e0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042512ac178e865847099e061b949402c0e0 2023-04-25 09:54:49,769 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425215fe8f4de244a878e8d23ad5a9f0485 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425215fe8f4de244a878e8d23ad5a9f0485 2023-04-25 09:54:49,770 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425235d5fa099774292bfaf148ebe4ef0b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425235d5fa099774292bfaf148ebe4ef0b1 2023-04-25 09:54:49,771 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252f316d92871d4ae494fe3b1e87f2d653 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252f316d92871d4ae494fe3b1e87f2d653 2023-04-25 09:54:49,772 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042533b9a67e8c5d4eec9a6cb4c2557a17b2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042533b9a67e8c5d4eec9a6cb4c2557a17b2 2023-04-25 09:54:49,773 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253cde8ea2f9bc4e9da61c85517c1ad7f1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253cde8ea2f9bc4e9da61c85517c1ad7f1 2023-04-25 09:54:49,774 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254b0f7ffb2e2c46c8810dd3e769c87db5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254b0f7ffb2e2c46c8810dd3e769c87db5 2023-04-25 09:54:49,775 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255d8839bc0dc84057b107c48b6c805c9c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255d8839bc0dc84057b107c48b6c805c9c 2023-04-25 09:54:49,776 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425662a93f9003b485db896b9f5f293c3b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425662a93f9003b485db896b9f5f293c3b6 2023-04-25 09:54:49,777 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425724623578b524f868317469e8024bd6d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425724623578b524f868317469e8024bd6d 2023-04-25 09:54:49,778 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257fb75cfa99ca4f3698275f99fbf6ccfd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257fb75cfa99ca4f3698275f99fbf6ccfd 2023-04-25 09:54:49,779 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258b3b1dc710d345e280a3b9dc1cda84e3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258b3b1dc710d345e280a3b9dc1cda84e3 2023-04-25 09:54:49,780 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425926b931890c44cf8ab2780f43280e2d2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425926b931890c44cf8ab2780f43280e2d2 2023-04-25 09:54:49,780 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259dd5c11559d44d3ea1f92f88a04ac4db to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304259dd5c11559d44d3ea1f92f88a04ac4db 2023-04-25 09:54:49,781 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b6c320ea587d495787801df4084207c2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425b6c320ea587d495787801df4084207c2 2023-04-25 09:54:49,782 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c7c9ea296f1a42559841cc3985bca888 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c7c9ea296f1a42559841cc3985bca888 2023-04-25 09:54:49,783 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d8c6e0c7ac744d37a16320a4144f7ecd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d8c6e0c7ac744d37a16320a4144f7ecd 2023-04-25 09:54:49,784 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425de7df2155409443faa46515d7fc96b9e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425de7df2155409443faa46515d7fc96b9e 2023-04-25 09:54:49,785 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e2dd56ce1c5f4df0b10279861a382d6a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e2dd56ce1c5f4df0b10279861a382d6a 2023-04-25 09:54:49,786 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ebf6b41b26e6449ca63ca83eba35e802 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ebf6b41b26e6449ca63ca83eba35e802 2023-04-25 09:54:49,787 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f0dd726b11cb4781bdeb0da7c2ac9090 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f0dd726b11cb4781bdeb0da7c2ac9090 2023-04-25 09:54:49,788 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f8dd7944206943c0bfc2f1d9d33a3b43 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f8dd7944206943c0bfc2f1d9d33a3b43 2023-04-25 09:54:49,789 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425fc89561b4ac643198709e201d85ef6c1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425fc89561b4ac643198709e201d85ef6c1 2023-04-25 09:54:49,789 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 09:54:49,791 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,793 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 09:54:49,795 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 09:54:49,796 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,796 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 09:54:49,796 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682416489796"}]},"ts":"9223372036854775807"} 2023-04-25 09:54:49,797 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 09:54:49,798 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5687fcb9918272e953b19ef4123710b6, NAME => 'TestAcidGuarantees,,1682416454675.5687fcb9918272e953b19ef4123710b6.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 09:54:49,798 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 09:54:49,798 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682416489798"}]},"ts":"9223372036854775807"} 2023-04-25 09:54:49,799 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 09:54:49,802 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-25 09:54:49,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-25 09:54:49,838 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-25 09:54:49,851 INFO [Listener at localhost/33705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 229), OpenFileDescriptor=518 (was 539), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=330 (was 392), ProcessCount=170 (was 170), AvailableMemoryMB=4116 (was 4225) 2023-04-25 09:54:49,862 INFO [Listener at localhost/33705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=518, MaxFileDescriptor=60000, SystemLoadAverage=330, ProcessCount=170, AvailableMemoryMB=4115 2023-04-25 09:54:49,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:54:49,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:54:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 09:54:49,866 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:54:49,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-25 09:54:49,867 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:54:49,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 09:54:49,868 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:49,869 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 empty. 2023-04-25 09:54:49,869 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:49,869 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:54:49,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 09:54:50,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 09:54:50,276 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:54:50,278 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1967ec64ec5d94eb49a819b814359794, NAME => 'TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:54:50,283 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:54:50,283 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1967ec64ec5d94eb49a819b814359794, disabling compactions & flushes 2023-04-25 09:54:50,283 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,283 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,283 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. after waiting 0 ms 2023-04-25 09:54:50,283 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,283 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,283 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:50,285 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:54:50,286 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416490285"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416490285"}]},"ts":"1682416490285"} 2023-04-25 09:54:50,286 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:54:50,287 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:54:50,287 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416490287"}]},"ts":"1682416490287"} 2023-04-25 09:54:50,288 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 09:54:50,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, ASSIGN}] 2023-04-25 09:54:50,292 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, ASSIGN 2023-04-25 09:54:50,293 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:54:50,444 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=1967ec64ec5d94eb49a819b814359794, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:50,444 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416490444"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416490444"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416490444"}]},"ts":"1682416490444"} 2023-04-25 09:54:50,445 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:54:50,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 09:54:50,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1967ec64ec5d94eb49a819b814359794, NAME => 'TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:54:50,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:54:50,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,602 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,603 DEBUG [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A 2023-04-25 09:54:50,603 DEBUG [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A 2023-04-25 09:54:50,603 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:50,604 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1967ec64ec5d94eb49a819b814359794 columnFamilyName A 2023-04-25 09:54:50,604 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.HStore(310): Store=1967ec64ec5d94eb49a819b814359794/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:50,604 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,605 DEBUG [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B 2023-04-25 09:54:50,605 DEBUG [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B 2023-04-25 09:54:50,606 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:50,606 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1967ec64ec5d94eb49a819b814359794 columnFamilyName B 2023-04-25 09:54:50,606 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.HStore(310): Store=1967ec64ec5d94eb49a819b814359794/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:50,606 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,607 DEBUG [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C 2023-04-25 09:54:50,607 DEBUG [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C 2023-04-25 09:54:50,607 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:54:50,607 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1967ec64ec5d94eb49a819b814359794 columnFamilyName C 2023-04-25 09:54:50,608 INFO [StoreOpener-1967ec64ec5d94eb49a819b814359794-1] regionserver.HStore(310): Store=1967ec64ec5d94eb49a819b814359794/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:54:50,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:54:50,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:50,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:54:50,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1967ec64ec5d94eb49a819b814359794; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74540787, jitterRate=0.11074428260326385}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:54:50,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:50,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., pid=30, masterSystemTime=1682416490597 2023-04-25 09:54:50,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:50,615 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=1967ec64ec5d94eb49a819b814359794, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:50,615 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416490615"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416490615"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416490615"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416490615"}]},"ts":"1682416490615"} 2023-04-25 09:54:50,617 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-25 09:54:50,617 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 in 171 msec 2023-04-25 09:54:50,619 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-25 09:54:50,619 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, ASSIGN in 327 msec 2023-04-25 09:54:50,619 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:54:50,619 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416490619"}]},"ts":"1682416490619"} 2023-04-25 09:54:50,620 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 09:54:50,622 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:54:50,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 758 msec 2023-04-25 09:54:50,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-25 09:54:50,970 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-25 09:54:50,971 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x22b4eae5 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:50,977 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12e82edd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:50,978 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:50,979 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:50,980 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:54:50,982 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:54:50,983 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:50,986 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:50,987 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:50,990 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:50,990 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:50,993 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:50,994 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:50,998 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:50,998 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:51,002 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:51,002 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:51,006 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:51,006 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:51,011 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:51,011 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:51,019 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:51,019 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:51,023 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:51,023 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:54:51,027 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:54:51,032 DEBUG [hconnection-0x51d9fe59-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,032 DEBUG [hconnection-0x22c1c582-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,032 DEBUG [hconnection-0x2935736a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:51,032 DEBUG [hconnection-0x4ce5b1af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,033 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,033 DEBUG [hconnection-0x308e5ac1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,033 DEBUG [hconnection-0x2d529ac3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,033 DEBUG [hconnection-0x6480942-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,033 DEBUG [hconnection-0x3d297f33-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,035 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,041 DEBUG [hconnection-0x230e2b26-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,041 DEBUG [hconnection-0x2a8df286-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:54:51,043 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39796, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,043 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,044 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,044 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,044 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39834, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,044 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,044 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,046 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39872, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:54:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:51,052 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:51,052 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:51,052 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:51,052 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:51,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:51,059 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:51,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:51,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:51,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:51,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:51,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:51,062 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:51,062 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:51,063 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 09:54:51,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:51,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:51,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:51,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:51,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:51,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:51,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:51,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:51,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416551112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416491091, max timestamp=1682416491114Num uniques 859; 2023-04-25 09:54:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416491091, max timestamp=1682416491114Num uniques 859; 2023-04-25 09:54:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:51,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:51,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416491091, max timestamp=1682416491114Num uniques 859; 2023-04-25 09:54:51,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:54:51,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416551127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416551128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416551129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416551130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416551217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416551231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416551233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416551234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416551234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416551421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416551434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416551436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416551436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416551439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,478 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/a2326a322fbf4134b035b3f6237f36f4 2023-04-25 09:54:51,490 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/855931fd5bfd4ec0a66663c483906b97 2023-04-25 09:54:51,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416551725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416551736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416551738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416551741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:51,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416551743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,901 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/90742755f7214efc957addd34ab26920 2023-04-25 09:54:51,906 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/a2326a322fbf4134b035b3f6237f36f4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/a2326a322fbf4134b035b3f6237f36f4 2023-04-25 09:54:51,910 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/a2326a322fbf4134b035b3f6237f36f4, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 09:54:51,911 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/855931fd5bfd4ec0a66663c483906b97 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/855931fd5bfd4ec0a66663c483906b97 2023-04-25 09:54:51,915 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/855931fd5bfd4ec0a66663c483906b97, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 09:54:51,915 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/90742755f7214efc957addd34ab26920 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/90742755f7214efc957addd34ab26920 2023-04-25 09:54:51,919 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/90742755f7214efc957addd34ab26920, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 09:54:51,920 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1967ec64ec5d94eb49a819b814359794 in 857ms, sequenceid=12, compaction requested=false 2023-04-25 09:54:51,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:51,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:51,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:51,923 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fffc9b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:51,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:51,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,926 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:51,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:51,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:51,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:51,931 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:51,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:51,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:51,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:51,931 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:51,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:51,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:51,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:51,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:51,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:51,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:51,941 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:51,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:51,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:51,941 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:51,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,042 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:52,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:52,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:52,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:52,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:52,045 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:52,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:52,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:52,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,047 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,047 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:52,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:52,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:52,049 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:52,050 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 09:54:52,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:52,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:52,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:52,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:52,051 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:52,051 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:52,059 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/01015e224e7644bb8c4e9675549e096d 2023-04-25 09:54:52,081 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/e02c1abe89254af6bf0b653c6bef46db 2023-04-25 09:54:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:52,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:52,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416552245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416552246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416552247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416552247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416552247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416552348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416552350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416552351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,498 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/dc164be2d4da409f96296db7a0e9cb22 2023-04-25 09:54:52,503 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/01015e224e7644bb8c4e9675549e096d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/01015e224e7644bb8c4e9675549e096d 2023-04-25 09:54:52,507 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/01015e224e7644bb8c4e9675549e096d, entries=150, sequenceid=46, filesize=11.6 K 2023-04-25 09:54:52,508 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/e02c1abe89254af6bf0b653c6bef46db as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/e02c1abe89254af6bf0b653c6bef46db 2023-04-25 09:54:52,512 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/e02c1abe89254af6bf0b653c6bef46db, entries=150, sequenceid=46, filesize=11.6 K 2023-04-25 09:54:52,512 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/dc164be2d4da409f96296db7a0e9cb22 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dc164be2d4da409f96296db7a0e9cb22 2023-04-25 09:54:52,516 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dc164be2d4da409f96296db7a0e9cb22, entries=150, sequenceid=46, filesize=11.6 K 2023-04-25 09:54:52,517 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 1967ec64ec5d94eb49a819b814359794 in 467ms, sequenceid=46, compaction requested=false 2023-04-25 09:54:52,517 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:52,517 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:52,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:52,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:52,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:52,519 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3177bb54[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:52,521 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:52,521 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,522 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,523 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:52,523 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:52,523 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,523 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:52,523 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:52,524 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:52,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:52,527 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:52,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:52,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:52,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:52,527 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:52,529 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,530 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,530 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,536 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,536 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:52,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:52,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,537 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:52,537 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:52,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 09:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:52,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416552613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416552613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416552616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,637 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:52,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:52,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:52,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:52,640 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:52,640 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,640 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:52,640 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:52,641 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,641 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,642 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,642 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:52,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:52,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:52,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:52,647 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:52,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@99053b6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:52,648 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:52,648 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,650 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:52,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:52,650 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:52,650 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,651 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:52,654 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:52,654 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:52,654 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:52,654 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:52,654 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:52,655 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,655 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,656 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,660 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:52,660 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:52,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:52,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416552717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416552717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416552719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,760 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:52,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:52,764 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:52,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:52,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:52,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:52,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:52,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:52,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:52,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:52,767 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:52,769 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21809c8b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:52,771 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:52,771 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,773 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:52,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:52,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,776 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:52,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:52,778 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:52,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:52,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:52,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:52,778 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:52,780 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,780 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,780 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,782 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,784 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,784 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,784 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,784 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:52,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:52,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,785 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:52,785 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:52,885 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:52,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:52,889 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:52,889 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,889 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:52,889 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:52,889 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,890 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:52,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:52,892 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:52,892 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:52,892 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:52,893 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,893 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:52,893 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:52,893 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:52,893 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:52,896 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:52,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f75a704[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:52,898 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:52,898 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,899 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,899 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:52,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:52,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:52,901 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:52,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:52,904 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:52,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:52,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:52,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:52,904 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:52,906 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,906 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,906 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,911 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:52,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:52,911 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:52,911 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:52,911 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:52,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:52,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416552919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416552920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416552921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:52,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/40a2dc06c5bb479a8edfa44f6e7f0ee3 2023-04-25 09:54:53,011 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:53,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:53,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:53,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:53,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:53,015 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:53,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:53,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:53,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,019 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:53,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:53,022 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:53,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:53,026 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:53,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:53,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@435408a5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:53,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,030 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:53,030 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:53,032 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:53,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:53,034 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:53,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:53,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:53,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:53,034 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:53,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,038 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:53,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,041 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:53,041 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,141 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:53,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:53,145 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,147 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:53,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:53,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:53,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:53,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:53,151 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4852ee9a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:53,152 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:53,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:53,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:53,155 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:53,157 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:53,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:53,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:53,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:53,157 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:53,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,162 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:53,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:53,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:53,165 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:53,165 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:53,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416553224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416553224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416553224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416553253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416553258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,265 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:53,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:53,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:53,269 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:53,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:53,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:53,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,271 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,271 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,271 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,271 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:53,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:53,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,275 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:53,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d1347df[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:53,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:53,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:53,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:53,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,280 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:53,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:53,283 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:53,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:53,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:53,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:53,283 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:53,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:53,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:53,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,289 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:53,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:53,289 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:53,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/efe3c16b9d214f6da00e597ab63ecfc7 2023-04-25 09:54:53,389 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:53,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:53,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:53,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:53,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:53,408 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:53,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:53,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:53,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/dfc7644feeea4f599fa9f2ba6704e4f8 2023-04-25 09:54:53,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,409 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,411 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,411 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,411 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:53,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:53,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:53,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/40a2dc06c5bb479a8edfa44f6e7f0ee3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/40a2dc06c5bb479a8edfa44f6e7f0ee3 2023-04-25 09:54:53,414 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,415 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:53,415 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:53,415 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:53,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:53,417 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,417 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,418 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,418 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/40a2dc06c5bb479a8edfa44f6e7f0ee3, entries=200, sequenceid=59, filesize=13.9 K 2023-04-25 09:54:53,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:53,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f3c80fe[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:53,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:53,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/efe3c16b9d214f6da00e597ab63ecfc7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/efe3c16b9d214f6da00e597ab63ecfc7 2023-04-25 09:54:53,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:53,422 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:53,422 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,422 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,424 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:53,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/efe3c16b9d214f6da00e597ab63ecfc7, entries=150, sequenceid=59, filesize=11.6 K 2023-04-25 09:54:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/dfc7644feeea4f599fa9f2ba6704e4f8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dfc7644feeea4f599fa9f2ba6704e4f8 2023-04-25 09:54:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:53,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:53,430 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:53,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:53,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:53,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:53,431 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:53,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:53,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dfc7644feeea4f599fa9f2ba6704e4f8, entries=150, sequenceid=59, filesize=11.6 K 2023-04-25 09:54:53,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,434 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:53,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1967ec64ec5d94eb49a819b814359794 in 882ms, sequenceid=59, compaction requested=true 2023-04-25 09:54:53,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:53,435 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:53,435 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:53,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:53,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:53,437 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:53,437 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:54:53,437 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,437 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/a2326a322fbf4134b035b3f6237f36f4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/01015e224e7644bb8c4e9675549e096d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/40a2dc06c5bb479a8edfa44f6e7f0ee3] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=37.1 K 2023-04-25 09:54:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:53,438 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:53,438 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:54:53,438 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a2326a322fbf4134b035b3f6237f36f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682416491059 2023-04-25 09:54:53,438 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,438 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/855931fd5bfd4ec0a66663c483906b97, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/e02c1abe89254af6bf0b653c6bef46db, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/efe3c16b9d214f6da00e597ab63ecfc7] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=34.8 K 2023-04-25 09:54:53,438 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 01015e224e7644bb8c4e9675549e096d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682416491122 2023-04-25 09:54:53,438 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 855931fd5bfd4ec0a66663c483906b97, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682416491059 2023-04-25 09:54:53,439 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 40a2dc06c5bb479a8edfa44f6e7f0ee3, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682416492242 2023-04-25 09:54:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,439 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e02c1abe89254af6bf0b653c6bef46db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682416491122 2023-04-25 09:54:53,440 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting efe3c16b9d214f6da00e597ab63ecfc7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682416492242 2023-04-25 09:54:53,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:53,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:53,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:53,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:53,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:53,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:53,441 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:53,441 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:53,449 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:53,451 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#232 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:53,467 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6c1ab76e40414f62b9cfaff2c6826e1a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6c1ab76e40414f62b9cfaff2c6826e1a 2023-04-25 09:54:53,480 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into 6c1ab76e40414f62b9cfaff2c6826e1a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:53,480 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:53,480 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=13, startTime=1682416493435; duration=0sec 2023-04-25 09:54:53,480 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:53,481 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:54:53,481 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:54:53,481 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:54:53,482 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,482 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/90742755f7214efc957addd34ab26920, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dc164be2d4da409f96296db7a0e9cb22, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dfc7644feeea4f599fa9f2ba6704e4f8] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=34.8 K 2023-04-25 09:54:53,482 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 90742755f7214efc957addd34ab26920, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682416491059 2023-04-25 09:54:53,482 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting dc164be2d4da409f96296db7a0e9cb22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682416491122 2023-04-25 09:54:53,483 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting dfc7644feeea4f599fa9f2ba6704e4f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682416492242 2023-04-25 09:54:53,488 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#233 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:53,542 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:53,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:53,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:53,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:53,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:53,545 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:53,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:53,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:53,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:53,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:53,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:53,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:53,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:53,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:53,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:53,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:53,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:53,549 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:53,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:53,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:53,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:53,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416553741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416553742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416553742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416553845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416553846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416553846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:53,864 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/797b8c7a572440589d352406439002ba as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/797b8c7a572440589d352406439002ba 2023-04-25 09:54:53,871 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into 797b8c7a572440589d352406439002ba(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:53,871 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:53,871 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=13, startTime=1682416493435; duration=0sec 2023-04-25 09:54:53,871 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:53,897 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/876141664e1c4cbe93be45c848028207 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/876141664e1c4cbe93be45c848028207 2023-04-25 09:54:53,901 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 876141664e1c4cbe93be45c848028207(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:53,901 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:53,901 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=13, startTime=1682416493435; duration=0sec 2023-04-25 09:54:53,901 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:53,957 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/ad030818889846649223a46d93e3e7e5 2023-04-25 09:54:54,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416554048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416554048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416554049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416554350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416554353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416554354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,369 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/19e6a489bcba465aa9e2235ade650647 2023-04-25 09:54:54,379 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/8e0853ea26234ee690e29fca920a015f 2023-04-25 09:54:54,383 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/ad030818889846649223a46d93e3e7e5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ad030818889846649223a46d93e3e7e5 2023-04-25 09:54:54,387 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ad030818889846649223a46d93e3e7e5, entries=150, sequenceid=79, filesize=11.6 K 2023-04-25 09:54:54,388 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/19e6a489bcba465aa9e2235ade650647 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e6a489bcba465aa9e2235ade650647 2023-04-25 09:54:54,393 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e6a489bcba465aa9e2235ade650647, entries=150, sequenceid=79, filesize=11.6 K 2023-04-25 09:54:54,394 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/8e0853ea26234ee690e29fca920a015f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/8e0853ea26234ee690e29fca920a015f 2023-04-25 09:54:54,398 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/8e0853ea26234ee690e29fca920a015f, entries=150, sequenceid=79, filesize=11.6 K 2023-04-25 09:54:54,399 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 1967ec64ec5d94eb49a819b814359794 in 850ms, sequenceid=79, compaction requested=false 2023-04-25 09:54:54,399 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:54,399 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:54,399 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:54,401 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cceb09c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:54,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:54,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,405 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:54,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:54,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:54,407 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:54,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:54,410 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:54,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:54,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:54,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:54,410 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:54,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:54,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:54,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,413 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,414 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:54,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:54,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:54,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:54,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,420 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:54,420 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:54,520 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:54,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:54,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:54,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:54,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:54,524 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:54,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:54,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:54,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:54,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:54,525 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:54,525 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:54,526 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,526 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:54,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:54,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:54,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:54,527 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:54,527 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:54,527 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:54,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:54,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416554897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416554897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416554897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:54,937 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/dc690389f6354da9a21d81fbdfa83b28 2023-04-25 09:54:54,947 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/4880238e58f44a56afe49fc6ed0b1571 2023-04-25 09:54:55,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416555001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416555001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416555001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,091 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 09:54:55,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416555204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416555204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416555204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416555259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,262 DEBUG [Thread-1787] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:55,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416555267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,269 DEBUG [Thread-1783] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:55,357 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/cf5e27d84ed34ccfa60a6b602fae6a08 2023-04-25 09:54:55,361 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/dc690389f6354da9a21d81fbdfa83b28 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/dc690389f6354da9a21d81fbdfa83b28 2023-04-25 09:54:55,364 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/dc690389f6354da9a21d81fbdfa83b28, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 09:54:55,365 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/4880238e58f44a56afe49fc6ed0b1571 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/4880238e58f44a56afe49fc6ed0b1571 2023-04-25 09:54:55,368 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/4880238e58f44a56afe49fc6ed0b1571, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 09:54:55,371 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/cf5e27d84ed34ccfa60a6b602fae6a08 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/cf5e27d84ed34ccfa60a6b602fae6a08 2023-04-25 09:54:55,375 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/cf5e27d84ed34ccfa60a6b602fae6a08, entries=150, sequenceid=95, filesize=11.6 K 2023-04-25 09:54:55,376 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 1967ec64ec5d94eb49a819b814359794 in 849ms, sequenceid=95, compaction requested=true 2023-04-25 09:54:55,376 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:55,376 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:55,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,378 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:55,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:55,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:55,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3337b5a1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:55,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:55,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,381 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:55,382 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:55,382 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,382 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:55,384 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:55,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:55,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:55,386 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:55,386 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:55,386 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:55,386 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:55,386 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:55,388 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,388 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,388 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:55,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,388 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:55,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:55,389 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:55,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:55,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:55,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:55,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,392 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:55,393 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:55,392 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:55,393 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:55,493 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:55,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:55,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:55,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:55,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:55,496 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:55,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:55,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:55,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:55,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:55,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:55,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:55,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:55,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:55,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:55,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:55,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:55,500 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:55,500 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:55,500 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:55,500 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 09:54:55,500 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:55,500 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:55,500 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:55,500 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:55,501 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:55,501 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:55,510 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/3c3d9f6f69824af396f53a3af9913211 2023-04-25 09:54:55,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:55,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:55,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416555519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416555520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416555521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416555623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416555624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416555624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416555826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416555827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416555828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:55,922 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/3b03156cc96e4b3eb0f01f43263b6d19 2023-04-25 09:54:56,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416556128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416556130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416556131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,332 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/6420a3b5d7cd4df9adcccf0130d76436 2023-04-25 09:54:56,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/3c3d9f6f69824af396f53a3af9913211 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3c3d9f6f69824af396f53a3af9913211 2023-04-25 09:54:56,344 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3c3d9f6f69824af396f53a3af9913211, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 09:54:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/3b03156cc96e4b3eb0f01f43263b6d19 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/3b03156cc96e4b3eb0f01f43263b6d19 2023-04-25 09:54:56,348 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/3b03156cc96e4b3eb0f01f43263b6d19, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 09:54:56,349 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/6420a3b5d7cd4df9adcccf0130d76436 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6420a3b5d7cd4df9adcccf0130d76436 2023-04-25 09:54:56,353 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6420a3b5d7cd4df9adcccf0130d76436, entries=150, sequenceid=115, filesize=11.6 K 2023-04-25 09:54:56,354 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1967ec64ec5d94eb49a819b814359794 in 854ms, sequenceid=115, compaction requested=true 2023-04-25 09:54:56,354 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:56,354 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:56,354 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:56,357 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,357 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:56,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:56,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:56,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:56,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ed21a7[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:56,359 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:56,359 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:56,360 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:56,361 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:56,361 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,361 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:56,365 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,365 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:56,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:56,367 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:56,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:56,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:56,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:56,368 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:56,370 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,370 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:56,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,370 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:56,371 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,374 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,374 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,374 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:56,374 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:56,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:56,374 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:56,374 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:56,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:56,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:56,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,375 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:56,375 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:56,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:56,475 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:56,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:56,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:56,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:56,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:56,479 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:56,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:56,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:56,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:56,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:56,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:56,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:56,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:56,482 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:56,482 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:56,483 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 09:54:56,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:56,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:56,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:56,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:56,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:56,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:56,491 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/3152217a209047dfa19b24a95d84e083 2023-04-25 09:54:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:56,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:56,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416556655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416556656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416556656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416556759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416556761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416556761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,901 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/35701dcf66a54d73bda6bc4f05ed9ce0 2023-04-25 09:54:56,910 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bbcda306cb714bf99afdbd49c16e9b37 2023-04-25 09:54:56,913 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/3152217a209047dfa19b24a95d84e083 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3152217a209047dfa19b24a95d84e083 2023-04-25 09:54:56,916 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3152217a209047dfa19b24a95d84e083, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 09:54:56,917 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/35701dcf66a54d73bda6bc4f05ed9ce0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/35701dcf66a54d73bda6bc4f05ed9ce0 2023-04-25 09:54:56,920 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/35701dcf66a54d73bda6bc4f05ed9ce0, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 09:54:56,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bbcda306cb714bf99afdbd49c16e9b37 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbcda306cb714bf99afdbd49c16e9b37 2023-04-25 09:54:56,924 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbcda306cb714bf99afdbd49c16e9b37, entries=150, sequenceid=127, filesize=11.6 K 2023-04-25 09:54:56,925 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1967ec64ec5d94eb49a819b814359794 in 442ms, sequenceid=127, compaction requested=true 2023-04-25 09:54:56,925 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:56,925 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:56,927 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:56,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:56,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:56,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3d72b4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:56,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:56,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,931 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:56,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:56,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:56,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:56,932 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:56,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:56,934 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:56,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:56,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:56,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:56,935 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:56,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:56,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:56,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:56,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:56,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:56,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:56,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:56,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:56,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:56,941 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:56,941 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:56,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:56,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:56,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-25 09:54:56,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416496651, max timestamp=1682416496965Num uniques 859; 2023-04-25 09:54:56,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416496651, max timestamp=1682416496965Num uniques 859; 2023-04-25 09:54:56,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416496651, max timestamp=1682416496965Num uniques 859; 2023-04-25 09:54:56,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:56,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:56,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:56,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:56,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/1e224460fc1142b598551267bb705922 2023-04-25 09:54:56,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/8af09b70aab84a9bb9669ce9d1e90c06 2023-04-25 09:54:57,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/6446cd8769ef42e6a29e71ee2838e516 2023-04-25 09:54:57,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416496988, max timestamp=1682416497004Num uniques 859; 2023-04-25 09:54:57,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:57,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416496988, max timestamp=1682416497004Num uniques 859; 2023-04-25 09:54:57,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:57,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416496988, max timestamp=1682416497004Num uniques 859; 2023-04-25 09:54:57,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:54:57,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/1e224460fc1142b598551267bb705922 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/1e224460fc1142b598551267bb705922 2023-04-25 09:54:57,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/1e224460fc1142b598551267bb705922, entries=150, sequenceid=149, filesize=11.8 K 2023-04-25 09:54:57,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/8af09b70aab84a9bb9669ce9d1e90c06 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8af09b70aab84a9bb9669ce9d1e90c06 2023-04-25 09:54:57,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8af09b70aab84a9bb9669ce9d1e90c06, entries=150, sequenceid=149, filesize=11.8 K 2023-04-25 09:54:57,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/6446cd8769ef42e6a29e71ee2838e516 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6446cd8769ef42e6a29e71ee2838e516 2023-04-25 09:54:57,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6446cd8769ef42e6a29e71ee2838e516, entries=150, sequenceid=149, filesize=11.8 K 2023-04-25 09:54:57,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=73.80 KB/75570 for 1967ec64ec5d94eb49a819b814359794 in 58ms, sequenceid=149, compaction requested=true 2023-04-25 09:54:57,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:57,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:57,024 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:54:57,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:57,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:57,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:57,024 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:54:57,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=80.51 KB heapSize=225.61 KB 2023-04-25 09:54:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:57,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:54:57,028 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:54:57,028 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:54:57,028 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,028 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6c1ab76e40414f62b9cfaff2c6826e1a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ad030818889846649223a46d93e3e7e5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/dc690389f6354da9a21d81fbdfa83b28, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3c3d9f6f69824af396f53a3af9913211, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3152217a209047dfa19b24a95d84e083, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/1e224460fc1142b598551267bb705922] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=69.9 K 2023-04-25 09:54:57,029 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6c1ab76e40414f62b9cfaff2c6826e1a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682416492242 2023-04-25 09:54:57,029 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:54:57,029 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:54:57,029 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,029 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/797b8c7a572440589d352406439002ba, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e6a489bcba465aa9e2235ade650647, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/4880238e58f44a56afe49fc6ed0b1571, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/3b03156cc96e4b3eb0f01f43263b6d19, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/35701dcf66a54d73bda6bc4f05ed9ce0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8af09b70aab84a9bb9669ce9d1e90c06] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=69.9 K 2023-04-25 09:54:57,030 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 797b8c7a572440589d352406439002ba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682416492242 2023-04-25 09:54:57,030 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ad030818889846649223a46d93e3e7e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682416492611 2023-04-25 09:54:57,030 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 19e6a489bcba465aa9e2235ade650647, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682416492611 2023-04-25 09:54:57,030 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting dc690389f6354da9a21d81fbdfa83b28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682416493740 2023-04-25 09:54:57,031 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4880238e58f44a56afe49fc6ed0b1571, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682416493740 2023-04-25 09:54:57,031 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3c3d9f6f69824af396f53a3af9913211, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682416494861 2023-04-25 09:54:57,031 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3b03156cc96e4b3eb0f01f43263b6d19, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682416494861 2023-04-25 09:54:57,031 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3152217a209047dfa19b24a95d84e083, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416495513 2023-04-25 09:54:57,032 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1e224460fc1142b598551267bb705922, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682416496655 2023-04-25 09:54:57,032 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 35701dcf66a54d73bda6bc4f05ed9ce0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416495513 2023-04-25 09:54:57,032 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8af09b70aab84a9bb9669ce9d1e90c06, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682416496655 2023-04-25 09:54:57,041 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,046 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#250 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:57,047 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#251 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:57,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,049 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,058 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416557056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,063 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,063 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/f45801e8aba449629903295d5297273a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f45801e8aba449629903295d5297273a 2023-04-25 09:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ee6d9f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,066 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/d75188555cbd493593495525dd11495f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d75188555cbd493593495525dd11495f 2023-04-25 09:54:57,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416557061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416557060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,068 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,068 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,069 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into f45801e8aba449629903295d5297273a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:57,069 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:57,069 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=10, startTime=1682416497024; duration=0sec 2023-04-25 09:54:57,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,069 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:57,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,069 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:54:57,071 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:54:57,071 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:54:57,071 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,071 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into d75188555cbd493593495525dd11495f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:57,071 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/876141664e1c4cbe93be45c848028207, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/8e0853ea26234ee690e29fca920a015f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/cf5e27d84ed34ccfa60a6b602fae6a08, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6420a3b5d7cd4df9adcccf0130d76436, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbcda306cb714bf99afdbd49c16e9b37, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6446cd8769ef42e6a29e71ee2838e516] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=69.9 K 2023-04-25 09:54:57,071 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:57,072 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=10, startTime=1682416497024; duration=0sec 2023-04-25 09:54:57,072 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:57,072 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 876141664e1c4cbe93be45c848028207, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682416492242 2023-04-25 09:54:57,072 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8e0853ea26234ee690e29fca920a015f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1682416492611 2023-04-25 09:54:57,074 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,074 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting cf5e27d84ed34ccfa60a6b602fae6a08, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682416493740 2023-04-25 09:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,075 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6420a3b5d7cd4df9adcccf0130d76436, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682416494861 2023-04-25 09:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,075 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting bbcda306cb714bf99afdbd49c16e9b37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416495513 2023-04-25 09:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,075 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6446cd8769ef42e6a29e71ee2838e516, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682416496655 2023-04-25 09:54:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:57,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,077 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,077 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,077 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,077 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,077 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,080 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,080 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,080 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,083 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,085 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,085 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,086 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#252 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:57,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416557162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416557168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416557168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,186 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,189 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,192 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,192 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,193 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,193 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fa5c5e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,199 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:57,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,203 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,203 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,213 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:57,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,213 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,313 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,317 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,321 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,321 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,321 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,321 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,322 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b7a2ff[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,324 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,324 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,330 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,330 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,332 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,332 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,332 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,337 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,337 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,338 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416557363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416557372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416557373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,438 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/269649c6d7484f0e8dee7f4899628efb 2023-04-25 09:54:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,441 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,443 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,444 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,444 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,445 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,445 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,445 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,445 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51dbca2f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,451 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,451 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,451 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/8d9e743f023a4c86b3dbf41de4ae9fca 2023-04-25 09:54:57,452 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,454 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,455 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,455 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,455 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,457 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,457 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,457 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,462 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,462 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:57,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,494 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/0ae4e7d4aa18450888d6039158ae07e3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0ae4e7d4aa18450888d6039158ae07e3 2023-04-25 09:54:57,498 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 0ae4e7d4aa18450888d6039158ae07e3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:57,498 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:57,498 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=10, startTime=1682416497024; duration=0sec 2023-04-25 09:54:57,498 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:57,563 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,566 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,570 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,570 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,570 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,570 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,579 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@149cd71e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,581 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,581 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,583 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,588 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,588 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,589 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,589 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,589 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,601 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,601 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,601 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416557666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416557675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:57,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416557676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,701 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,704 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,706 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,708 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,708 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,708 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,708 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,710 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e6e717f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,713 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,717 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,718 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,721 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,724 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,724 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,724 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:57,825 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,828 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,832 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,832 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,832 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:57,832 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,836 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:57,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e68417[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:57,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:57,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:57,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:57,841 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:57,843 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:57,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:57,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:57,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:57,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:57,845 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,845 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,845 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:57,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:57,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:57,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,851 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:57,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:57,852 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:57,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:57,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/0e60e610cf2e44478b6c2ef280d52d71 2023-04-25 09:54:57,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/269649c6d7484f0e8dee7f4899628efb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/269649c6d7484f0e8dee7f4899628efb 2023-04-25 09:54:57,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/269649c6d7484f0e8dee7f4899628efb, entries=200, sequenceid=178, filesize=14.1 K 2023-04-25 09:54:57,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/8d9e743f023a4c86b3dbf41de4ae9fca as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8d9e743f023a4c86b3dbf41de4ae9fca 2023-04-25 09:54:57,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8d9e743f023a4c86b3dbf41de4ae9fca, entries=150, sequenceid=178, filesize=11.8 K 2023-04-25 09:54:57,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/0e60e610cf2e44478b6c2ef280d52d71 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0e60e610cf2e44478b6c2ef280d52d71 2023-04-25 09:54:57,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0e60e610cf2e44478b6c2ef280d52d71, entries=150, sequenceid=178, filesize=11.8 K 2023-04-25 09:54:57,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~225.56 KB/230976, currentSize=100.63 KB/103050 for 1967ec64ec5d94eb49a819b814359794 in 858ms, sequenceid=178, compaction requested=false 2023-04-25 09:54:57,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:57,952 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:57,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:57,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:57,955 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:57,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:57,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:57,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,957 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:57,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:57,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:57,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:57,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:57,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:57,959 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:57,959 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:57,959 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:54:57,959 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:57,959 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:57,959 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:57,960 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:57,960 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:57,960 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:57,970 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6f227990957d4ecf8b61d56f822db793 2023-04-25 09:54:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:58,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416558212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416558212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416558212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416558330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416558330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416558330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,382 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/00f34e2080b249468e7a07f7d21ba222 2023-04-25 09:54:58,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416558532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416558533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416558534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,806 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bbdd15181fbf4f64a98e6cc92d203688 2023-04-25 09:54:58,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6f227990957d4ecf8b61d56f822db793 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6f227990957d4ecf8b61d56f822db793 2023-04-25 09:54:58,817 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6f227990957d4ecf8b61d56f822db793, entries=150, sequenceid=199, filesize=11.8 K 2023-04-25 09:54:58,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/00f34e2080b249468e7a07f7d21ba222 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/00f34e2080b249468e7a07f7d21ba222 2023-04-25 09:54:58,822 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/00f34e2080b249468e7a07f7d21ba222, entries=150, sequenceid=199, filesize=11.8 K 2023-04-25 09:54:58,823 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bbdd15181fbf4f64a98e6cc92d203688 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbdd15181fbf4f64a98e6cc92d203688 2023-04-25 09:54:58,826 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbdd15181fbf4f64a98e6cc92d203688, entries=150, sequenceid=199, filesize=11.8 K 2023-04-25 09:54:58,827 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1967ec64ec5d94eb49a819b814359794 in 868ms, sequenceid=199, compaction requested=true 2023-04-25 09:54:58,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:58,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:58,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:58,829 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:58,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e3b656[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:58,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:58,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:58,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:58,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:58,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:58,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:54:58,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:58,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:58,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:58,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:58,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:58,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:58,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:58,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:58,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:58,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:58,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:58,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:58,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:58,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:58,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:58,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:58,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:58,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,848 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:58,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:58,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:58,851 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:58,851 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:58,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:58,852 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:58,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:58,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/f6070f68796a45abb187e2c1962ceb87 2023-04-25 09:54:58,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416558859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416558860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416558863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,875 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/2b69a13fc0244e5aa7f627dc6906845b 2023-04-25 09:54:58,952 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:58,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:58,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:58,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:58,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:58,955 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:58,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:58,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:58,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:58,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:58,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:58,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:58,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:58,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:58,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:58,963 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:58,963 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:58,964 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:58,964 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:58,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416558964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416558964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,966 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,967 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:58,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb5ae53[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:58,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:58,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:58,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416558967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,969 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:58,970 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,970 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:58,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:58,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:58,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:58,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:58,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:58,972 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:58,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:58,974 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:58,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:58,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:58,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:58,975 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:58,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:58,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:58,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:58,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:58,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:58,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:58,982 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:58,983 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:58,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:58,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:58,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:58,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,083 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,086 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,086 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,086 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,086 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,086 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,090 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,090 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,090 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,090 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@265dd7d3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,095 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,097 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,100 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,100 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,103 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:59,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,106 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,106 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416559167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416559168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416559170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,206 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f93933[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,218 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,220 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,223 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,223 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,225 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,225 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,229 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,230 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,230 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:59,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/95a897008a364dc8ad7729a86146bf36 2023-04-25 09:54:59,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/f6070f68796a45abb187e2c1962ceb87 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f6070f68796a45abb187e2c1962ceb87 2023-04-25 09:54:59,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416559289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,292 DEBUG [Thread-1787] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:54:59,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f6070f68796a45abb187e2c1962ceb87, entries=150, sequenceid=215, filesize=11.8 K 2023-04-25 09:54:59,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/2b69a13fc0244e5aa7f627dc6906845b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2b69a13fc0244e5aa7f627dc6906845b 2023-04-25 09:54:59,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2b69a13fc0244e5aa7f627dc6906845b, entries=150, sequenceid=215, filesize=11.8 K 2023-04-25 09:54:59,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/95a897008a364dc8ad7729a86146bf36 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/95a897008a364dc8ad7729a86146bf36 2023-04-25 09:54:59,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/95a897008a364dc8ad7729a86146bf36, entries=150, sequenceid=215, filesize=11.8 K 2023-04-25 09:54:59,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 1967ec64ec5d94eb49a819b814359794 in 467ms, sequenceid=215, compaction requested=true 2023-04-25 09:54:59,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:59,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:59,305 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:59,305 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:59,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:59,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:59,306 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50842 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:59,306 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:54:59,306 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,306 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f45801e8aba449629903295d5297273a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/269649c6d7484f0e8dee7f4899628efb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6f227990957d4ecf8b61d56f822db793, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f6070f68796a45abb187e2c1962ceb87] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=49.7 K 2023-04-25 09:54:59,306 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:59,306 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f45801e8aba449629903295d5297273a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682416496655 2023-04-25 09:54:59,306 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:54:59,307 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,307 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d75188555cbd493593495525dd11495f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8d9e743f023a4c86b3dbf41de4ae9fca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/00f34e2080b249468e7a07f7d21ba222, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2b69a13fc0244e5aa7f627dc6906845b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=47.3 K 2023-04-25 09:54:59,307 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 269649c6d7484f0e8dee7f4899628efb, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682416497010 2023-04-25 09:54:59,307 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d75188555cbd493593495525dd11495f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682416496655 2023-04-25 09:54:59,307 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6f227990957d4ecf8b61d56f822db793, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682416497047 2023-04-25 09:54:59,307 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8d9e743f023a4c86b3dbf41de4ae9fca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682416497018 2023-04-25 09:54:59,307 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f6070f68796a45abb187e2c1962ceb87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416498184 2023-04-25 09:54:59,308 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 00f34e2080b249468e7a07f7d21ba222, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682416497047 2023-04-25 09:54:59,308 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2b69a13fc0244e5aa7f627dc6906845b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416498184 2023-04-25 09:54:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:54:59,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:54:59,309 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:54:59,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:59,309 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:54:59,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:59,309 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:54:59,309 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:54:59,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/8142693760fd459687e8e35154c77b17 2023-04-25 09:54:59,319 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#262 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:59,325 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#263 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:59,330 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,336 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,337 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,337 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,338 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,338 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,338 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,338 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,340 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,340 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,340 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,340 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,341 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/63780008ba7f48d2a16ff7d4c7fee19e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/63780008ba7f48d2a16ff7d4c7fee19e 2023-04-25 09:54:59,345 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into 63780008ba7f48d2a16ff7d4c7fee19e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:59,345 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:59,345 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=12, startTime=1682416499304; duration=0sec 2023-04-25 09:54:59,345 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:54:59,345 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:54:59,346 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:54:59,346 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:54:59,346 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,346 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0ae4e7d4aa18450888d6039158ae07e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0e60e610cf2e44478b6c2ef280d52d71, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbdd15181fbf4f64a98e6cc92d203688, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/95a897008a364dc8ad7729a86146bf36] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=47.3 K 2023-04-25 09:54:59,346 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0ae4e7d4aa18450888d6039158ae07e3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682416496655 2023-04-25 09:54:59,347 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0e60e610cf2e44478b6c2ef280d52d71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1682416497018 2023-04-25 09:54:59,347 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting bbdd15181fbf4f64a98e6cc92d203688, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682416497047 2023-04-25 09:54:59,347 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 95a897008a364dc8ad7729a86146bf36, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416498184 2023-04-25 09:54:59,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c76fc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,358 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,358 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,358 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,358 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,358 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,361 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:54:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,364 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,367 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,368 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,368 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416559366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,468 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,471 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,471 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,472 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,472 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416559470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39844 deadline: 1682416559471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416559473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,475 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,475 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,475 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,475 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39834 deadline: 1682416559475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,477 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,477 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad8794[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,484 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,487 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,487 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,489 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,490 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:59,493 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,493 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,493 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,593 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,597 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,599 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,599 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,599 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,599 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,601 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,601 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,601 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,601 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e062bdd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,607 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,607 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,607 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,612 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,612 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,612 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,612 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,612 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,618 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,618 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,618 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:59,619 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,619 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,618 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416559674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,719 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,723 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,725 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,727 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,727 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,727 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,727 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,729 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@433bb38f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,734 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,734 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,734 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,734 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,734 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,734 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/0bc77b1942f144c28fd1b364dad87162 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/0bc77b1942f144c28fd1b364dad87162 2023-04-25 09:54:59,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/74e38d9fd37a4f818384c7cfb07ff557 2023-04-25 09:54:59,739 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into 0bc77b1942f144c28fd1b364dad87162(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:59,739 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:59,739 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=12, startTime=1682416499305; duration=0sec 2023-04-25 09:54:59,739 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,742 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,743 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,745 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,745 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,745 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,746 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,748 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,750 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,751 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,751 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,771 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/146b09fd854c4e18ae84b3a89c572a85 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/146b09fd854c4e18ae84b3a89c572a85 2023-04-25 09:54:59,779 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 146b09fd854c4e18ae84b3a89c572a85(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:54:59,779 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:54:59,779 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=12, startTime=1682416499305; duration=0sec 2023-04-25 09:54:59,779 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:54:59,851 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,854 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,856 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,858 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,858 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,858 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,858 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2083f682[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,863 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,863 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,866 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,866 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,866 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:54:59,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:54:59,868 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:54:59,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:54:59,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:54:59,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:54:59,868 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:54:59,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,871 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,872 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:54:59,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:54:59,875 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:54:59,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,876 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:54:59,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:54:59,876 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:54:59,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:54:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:54:59,976 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:54:59,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416559976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:54:59,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416559977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:54:59,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:54:59,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416559978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:54:59,979 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:54:59,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:54:59,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:54:59,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:54:59,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416559980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,981 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:54:59,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:54:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:54:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:54:59,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:54:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:54:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:54:59,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:54:59,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:54:59,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:54:59,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:54:59,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:54:59,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:54:59,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:54:59,985 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:54:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:54:59,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:54:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:54:59,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:54:59,987 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:54:59,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a784421[Count = 0] remaining members to acquire global barrier 2023-04-25 09:54:59,987 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:54:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:54:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:54:59,989 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:54:59,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:54:59,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:54:59,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:54:59,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:54:59,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:54:59,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:00,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:00,002 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:00,002 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:00,002 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:00,002 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:00,002 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:00,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:00,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:00,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:00,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:00,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,007 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:00,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:00,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:00,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:00,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,008 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:00,008 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:00,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:00,108 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:00,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:00,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:00,112 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:00,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:00,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:00,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:00,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:00,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:00,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:00,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:00,115 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:00,116 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:00,116 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:00,116 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:00,116 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:00,116 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:00,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:00,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,120 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:00,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:00,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355fd670[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:00,121 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:00,121 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,123 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:00,123 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:00,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:00,124 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:00,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,125 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:00,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:00,128 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:00,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:00,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:00,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:00,128 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:00,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:00,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:00,138 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,139 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:00,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:00,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:00,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:00,141 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,141 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:00,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:00,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:00,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/c3f0c777fd0647369cbb69b81e9a876a 2023-04-25 09:55:00,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/8142693760fd459687e8e35154c77b17 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/8142693760fd459687e8e35154c77b17 2023-04-25 09:55:00,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/8142693760fd459687e8e35154c77b17, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 09:55:00,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/74e38d9fd37a4f818384c7cfb07ff557 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/74e38d9fd37a4f818384c7cfb07ff557 2023-04-25 09:55:00,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/74e38d9fd37a4f818384c7cfb07ff557, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 09:55:00,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/c3f0c777fd0647369cbb69b81e9a876a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c3f0c777fd0647369cbb69b81e9a876a 2023-04-25 09:55:00,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c3f0c777fd0647369cbb69b81e9a876a, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 09:55:00,166 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 1967ec64ec5d94eb49a819b814359794 in 858ms, sequenceid=232, compaction requested=false 2023-04-25 09:55:00,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:00,242 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:00,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:00,245 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:00,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:00,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:00,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:00,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:00,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:00,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:00,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:00,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:00,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:00,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:00,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:00,249 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:00,249 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:00,250 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:55:00,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:00,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:00,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:00,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:00,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:00,250 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:00,259 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/0a424ff9ca09425b8a70282ccfb0f326 2023-04-25 09:55:00,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:00,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:00,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:00,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416560514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:00,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416560617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,677 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/8985b0fbf3ab4584b2f84b830f2c7e37 2023-04-25 09:55:00,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416560821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:00,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416560978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:00,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39820 deadline: 1682416560986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:00,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:00,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416560990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,087 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/5ba436dc9a7148eaad9105dd7e9d71b1 2023-04-25 09:55:01,091 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/0a424ff9ca09425b8a70282ccfb0f326 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/0a424ff9ca09425b8a70282ccfb0f326 2023-04-25 09:55:01,095 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/0a424ff9ca09425b8a70282ccfb0f326, entries=150, sequenceid=250, filesize=11.8 K 2023-04-25 09:55:01,095 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/8985b0fbf3ab4584b2f84b830f2c7e37 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8985b0fbf3ab4584b2f84b830f2c7e37 2023-04-25 09:55:01,099 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8985b0fbf3ab4584b2f84b830f2c7e37, entries=150, sequenceid=250, filesize=11.8 K 2023-04-25 09:55:01,100 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/5ba436dc9a7148eaad9105dd7e9d71b1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/5ba436dc9a7148eaad9105dd7e9d71b1 2023-04-25 09:55:01,105 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/5ba436dc9a7148eaad9105dd7e9d71b1, entries=150, sequenceid=250, filesize=11.8 K 2023-04-25 09:55:01,106 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 1967ec64ec5d94eb49a819b814359794 in 856ms, sequenceid=250, compaction requested=true 2023-04-25 09:55:01,106 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:01,106 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:01,110 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b66bc9f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:01,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:01,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:01,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:01,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:01,114 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:01,118 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:01,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:01,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:01,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:01,118 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:01,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,120 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:01,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:01,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,125 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:01,125 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:01,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:01,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:01,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:55:01,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:01,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:01,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:01,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:01,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:01,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:01,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/5525c0aa2a56451dba26b162d3804f70 2023-04-25 09:55:01,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/39dba5ffd72041dcbc0f83194e371fe5 2023-04-25 09:55:01,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:01,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416561162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,225 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:01,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:01,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:01,228 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:01,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:01,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:01,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,230 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,230 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,230 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:01,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:01,231 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:01,232 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146e7897[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:01,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,236 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:01,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:01,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:01,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:01,241 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:01,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:01,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:01,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:01,241 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:01,243 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,243 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,243 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,244 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:01,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:01,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,248 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:01,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,248 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:01,248 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:01,248 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,248 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:01,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416561265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,348 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:01,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:01,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:01,351 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:01,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:01,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:01,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:01,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:01,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:01,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:01,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:01,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:01,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:01,355 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:01,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:01,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:01,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:01,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:01,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:01,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:01,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbf7f77[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:01,360 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:01,360 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,362 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:01,362 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:01,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:01,363 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,363 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,363 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,363 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:01,366 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:01,366 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:01,366 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:01,366 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:01,366 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:01,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,369 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:01,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,375 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,375 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,375 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,375 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,376 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:01,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416561470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,475 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:01,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:01,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:01,479 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:01,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:01,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:01,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:01,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:01,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:01,482 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,482 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:01,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:01,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:01,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:01,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:01,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5798c6c6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:01,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:01,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:01,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:01,488 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,489 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:01,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:01,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,490 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,490 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:01,492 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:01,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:01,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:01,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:01,492 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:01,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,495 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:01,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:01,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:01,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:01,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:01,499 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:01,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:01,499 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:01,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/67b47c40a22941b08b0312da1ab6118b 2023-04-25 09:55:01,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/5525c0aa2a56451dba26b162d3804f70 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5525c0aa2a56451dba26b162d3804f70 2023-04-25 09:55:01,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5525c0aa2a56451dba26b162d3804f70, entries=200, sequenceid=268, filesize=14.3 K 2023-04-25 09:55:01,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/39dba5ffd72041dcbc0f83194e371fe5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/39dba5ffd72041dcbc0f83194e371fe5 2023-04-25 09:55:01,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/39dba5ffd72041dcbc0f83194e371fe5, entries=150, sequenceid=268, filesize=11.9 K 2023-04-25 09:55:01,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/67b47c40a22941b08b0312da1ab6118b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/67b47c40a22941b08b0312da1ab6118b 2023-04-25 09:55:01,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/67b47c40a22941b08b0312da1ab6118b, entries=150, sequenceid=268, filesize=11.9 K 2023-04-25 09:55:01,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1967ec64ec5d94eb49a819b814359794 in 445ms, sequenceid=268, compaction requested=true 2023-04-25 09:55:01,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:01,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:01,572 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:01,573 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:01,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:01,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:01,574 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51178 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:01,574 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48738 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:01,574 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:55:01,574 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:55:01,574 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,574 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,574 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/0bc77b1942f144c28fd1b364dad87162, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/74e38d9fd37a4f818384c7cfb07ff557, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8985b0fbf3ab4584b2f84b830f2c7e37, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/39dba5ffd72041dcbc0f83194e371fe5] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=47.6 K 2023-04-25 09:55:01,574 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/63780008ba7f48d2a16ff7d4c7fee19e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/8142693760fd459687e8e35154c77b17, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/0a424ff9ca09425b8a70282ccfb0f326, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5525c0aa2a56451dba26b162d3804f70] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=50.0 K 2023-04-25 09:55:01,574 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0bc77b1942f144c28fd1b364dad87162, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416498184 2023-04-25 09:55:01,574 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 63780008ba7f48d2a16ff7d4c7fee19e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416498184 2023-04-25 09:55:01,575 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 74e38d9fd37a4f818384c7cfb07ff557, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682416498857 2023-04-25 09:55:01,575 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8142693760fd459687e8e35154c77b17, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682416498857 2023-04-25 09:55:01,575 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8985b0fbf3ab4584b2f84b830f2c7e37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682416499338 2023-04-25 09:55:01,575 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0a424ff9ca09425b8a70282ccfb0f326, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682416499338 2023-04-25 09:55:01,575 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 39dba5ffd72041dcbc0f83194e371fe5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682416500506 2023-04-25 09:55:01,575 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5525c0aa2a56451dba26b162d3804f70, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682416500504 2023-04-25 09:55:01,583 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:01,585 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:01,599 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:01,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:01,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:01,602 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:01,602 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/549e7085fed44fdea1cc365b3051c9ab as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/549e7085fed44fdea1cc365b3051c9ab 2023-04-25 09:55:01,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:01,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:01,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:01,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:01,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:01,606 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into 549e7085fed44fdea1cc365b3051c9ab(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:01,606 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:01,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:01,606 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=12, startTime=1682416501572; duration=0sec 2023-04-25 09:55:01,607 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:01,607 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:01,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:01,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:01,608 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48738 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:01,608 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:55:01,608 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:55:01,608 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:01,608 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/146b09fd854c4e18ae84b3a89c572a85, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c3f0c777fd0647369cbb69b81e9a876a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/5ba436dc9a7148eaad9105dd7e9d71b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/67b47c40a22941b08b0312da1ab6118b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=47.6 K 2023-04-25 09:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:01,609 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:01,609 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:01,609 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:01,609 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 146b09fd854c4e18ae84b3a89c572a85, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416498184 2023-04-25 09:55:01,609 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:01,609 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:01,609 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c3f0c777fd0647369cbb69b81e9a876a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682416498857 2023-04-25 09:55:01,609 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5ba436dc9a7148eaad9105dd7e9d71b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682416499338 2023-04-25 09:55:01,610 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 67b47c40a22941b08b0312da1ab6118b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682416500506 2023-04-25 09:55:01,617 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/ec8583843f2f456b89fbb6815ffbb966 2023-04-25 09:55:01,618 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:01,633 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/43a4d92190fc4e9d83f9d762f1d6b512 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/43a4d92190fc4e9d83f9d762f1d6b512 2023-04-25 09:55:01,639 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 43a4d92190fc4e9d83f9d762f1d6b512(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:01,639 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:01,639 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=12, startTime=1682416501573; duration=0sec 2023-04-25 09:55:01,639 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:01,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:01,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:01,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:01,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416561812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:01,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:01,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416561915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,002 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/6bbd4aa269694b3481f1c9337fd7f789 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/6bbd4aa269694b3481f1c9337fd7f789 2023-04-25 09:55:02,007 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into 6bbd4aa269694b3481f1c9337fd7f789(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:02,007 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:02,007 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=12, startTime=1682416501572; duration=0sec 2023-04-25 09:55:02,007 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:02,042 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/65a10b22113a4bb89d4f8100d4d2801a 2023-04-25 09:55:02,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:02,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416562118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:02,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39816 deadline: 1682416562422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,452 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/042435aef1e2453da5bd821efd342268 2023-04-25 09:55:02,456 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/ec8583843f2f456b89fbb6815ffbb966 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ec8583843f2f456b89fbb6815ffbb966 2023-04-25 09:55:02,460 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ec8583843f2f456b89fbb6815ffbb966, entries=150, sequenceid=283, filesize=11.9 K 2023-04-25 09:55:02,460 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/65a10b22113a4bb89d4f8100d4d2801a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/65a10b22113a4bb89d4f8100d4d2801a 2023-04-25 09:55:02,464 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/65a10b22113a4bb89d4f8100d4d2801a, entries=150, sequenceid=283, filesize=11.9 K 2023-04-25 09:55:02,464 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/042435aef1e2453da5bd821efd342268 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/042435aef1e2453da5bd821efd342268 2023-04-25 09:55:02,468 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/042435aef1e2453da5bd821efd342268, entries=150, sequenceid=283, filesize=11.9 K 2023-04-25 09:55:02,468 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1967ec64ec5d94eb49a819b814359794 in 860ms, sequenceid=283, compaction requested=false 2023-04-25 09:55:02,468 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:02,468 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:02,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:02,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:02,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:02,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,471 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49841aee[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:02,472 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:02,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:02,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:02,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:02,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:02,478 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:02,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:02,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:02,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:02,478 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:02,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:02,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:02,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:02,485 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:02,485 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:02,585 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:02,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:02,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:02,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:02,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:02,589 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:02,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:02,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:02,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:02,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:02,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:02,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:02,592 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:55:02,592 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:02,593 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:02,593 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:02,593 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:02,593 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:02,593 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:02,601 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/cb56223cdf644249a2aa5e695481d6b2 2023-04-25 09:55:02,611 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/498851ac49aa497ca82e08619f65cd40 2023-04-25 09:55:02,621 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/e49ed3eaf4ad4289b77dce9e60604003 2023-04-25 09:55:02,625 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/cb56223cdf644249a2aa5e695481d6b2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/cb56223cdf644249a2aa5e695481d6b2 2023-04-25 09:55:02,628 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/cb56223cdf644249a2aa5e695481d6b2, entries=150, sequenceid=303, filesize=11.9 K 2023-04-25 09:55:02,629 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/498851ac49aa497ca82e08619f65cd40 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/498851ac49aa497ca82e08619f65cd40 2023-04-25 09:55:02,633 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/498851ac49aa497ca82e08619f65cd40, entries=150, sequenceid=303, filesize=11.9 K 2023-04-25 09:55:02,635 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/e49ed3eaf4ad4289b77dce9e60604003 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/e49ed3eaf4ad4289b77dce9e60604003 2023-04-25 09:55:02,639 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/e49ed3eaf4ad4289b77dce9e60604003, entries=150, sequenceid=303, filesize=11.9 K 2023-04-25 09:55:02,640 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 1967ec64ec5d94eb49a819b814359794 in 48ms, sequenceid=303, compaction requested=true 2023-04-25 09:55:02,640 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:02,640 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:02,641 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,641 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:02,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec97989[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:02,644 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:02,645 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,646 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,647 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,647 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:02,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:02,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:02,647 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:02,648 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:02,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:02,652 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:02,652 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:02,654 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,654 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,654 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:02,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,658 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,659 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:02,659 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:02,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:02,759 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:02,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:02,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:02,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:02,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:02,762 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:02,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:02,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:02,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:02,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,767 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,768 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,768 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:02,768 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:02,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:02,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:02,770 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,770 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:02,770 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:02,770 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:02,770 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:02,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,773 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:02,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496aefe0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:02,774 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:02,774 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,776 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:02,776 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:02,776 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:02,776 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,776 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:02,778 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,778 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:02,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:02,781 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:02,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:02,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:02,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:02,781 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:02,783 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,783 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,783 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:02,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:02,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:02,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,788 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:02,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,788 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:02,888 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:02,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:02,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:02,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:02,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:02,892 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:02,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:02,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:02,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:02,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,894 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:02,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:02,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:02,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:02,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:02,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:02,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:02,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:02,896 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:02,896 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,896 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:02,897 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:02,897 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:02,899 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d5befd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:02,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:02,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,903 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:02,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:02,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:02,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,905 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:02,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:02,908 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:02,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:02,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:02,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:02,908 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:02,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:02,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:02,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,911 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,912 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,915 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:02,915 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:02,915 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:02,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:02,915 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:02,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:02,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:02,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:02,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,916 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:02,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:02,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:02,916 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:02,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:02,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:02,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:02,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:02,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:02,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:02,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:02,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:02,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:02,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416502984, max timestamp=1682416502991Num uniques 859; 2023-04-25 09:55:02,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416502984, max timestamp=1682416502991Num uniques 859; 2023-04-25 09:55:02,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:02,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:02,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416502984, max timestamp=1682416502991Num uniques 859; 2023-04-25 09:55:02,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:03,016 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682416503001, max timestamp=1682416503013Num uniques 1018; 2023-04-25 09:55:03,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-25 09:55:03,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682416503001, max timestamp=1682416503013Num uniques 1018; 2023-04-25 09:55:03,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682416503001, max timestamp=1682416503013Num uniques 1018; 2023-04-25 09:55:03,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-25 09:55:03,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-25 09:55:03,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,020 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,021 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,021 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,023 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,023 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,025 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,025 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,025 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,025 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,027 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31ef8c63[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,030 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,030 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,031 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,031 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416563033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416563035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416563034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,040 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,040 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,042 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416563038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,043 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,046 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,046 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,046 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:03,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416563139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416563140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416563141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416563144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,146 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,149 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,151 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,152 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,152 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@540fbee6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,165 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,165 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,168 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,173 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,173 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,173 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,173 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,173 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,273 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,277 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,281 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,281 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,281 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,281 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,283 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a045e68[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,291 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,291 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,292 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,292 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,297 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,297 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,297 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,297 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:03,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,298 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,298 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416563343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416563343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416563344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/4d73de1c7ae2473b8d63e42d28b1ff54 2023-04-25 09:55:03,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416563347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,398 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,402 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fd33383[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,421 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,423 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,423 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,425 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,425 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,425 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,425 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,425 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:03,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,433 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,433 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,433 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,533 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,537 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,541 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,541 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,541 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,541 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,542 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,544 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72122c8e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,544 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,547 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,547 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,547 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,547 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,550 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,553 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,553 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,555 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,555 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,555 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,555 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,559 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,559 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416563646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416563646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416563649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416563652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,659 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,662 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,665 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,665 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,666 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,666 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,669 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4354a4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,672 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,676 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,677 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,680 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,683 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,684 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,684 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/ead6287b6d7f44e799e95371705bef41 2023-04-25 09:55:03,784 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,787 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,792 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,793 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,793 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,794 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,795 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,795 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,795 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,797 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,797 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,797 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335ca6ba[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,801 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,802 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,805 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,805 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,807 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,808 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,812 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,812 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,912 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:03,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:03,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:03,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:03,915 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:03,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:03,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:03,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,921 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,921 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,921 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:03,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:03,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:03,922 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,922 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:03,923 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:03,923 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:03,925 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,925 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,926 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:03,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b5c479[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:03,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:03,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:03,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:03,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,929 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:03,932 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:03,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:03,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:03,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:03,932 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:03,938 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,938 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:03,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,938 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:03,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:03,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:03,943 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:03,942 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:03,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:03,943 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:03,943 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:03,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:03,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,043 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:04,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:04,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:04,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:04,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:04,046 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:04,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:04,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:04,047 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,047 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:04,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:04,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:04,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:04,050 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:04,058 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7620fc1d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:04,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:04,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:04,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:04,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:04,061 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,061 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:04,063 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:04,065 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:04,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:04,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:04,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:04,065 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:04,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,068 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:04,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,073 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:04,073 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:04,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:04,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416564150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:04,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416564152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:04,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416564153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:04,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416564158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/823d721dffa644c98aebb50247404853 2023-04-25 09:55:04,173 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:04,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/4d73de1c7ae2473b8d63e42d28b1ff54 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d73de1c7ae2473b8d63e42d28b1ff54 2023-04-25 09:55:04,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:04,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:04,176 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:04,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:04,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:04,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d73de1c7ae2473b8d63e42d28b1ff54, entries=150, sequenceid=315, filesize=11.9 K 2023-04-25 09:55:04,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:04,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:04,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:04,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/ead6287b6d7f44e799e95371705bef41 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/ead6287b6d7f44e799e95371705bef41 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:04,180 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,180 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:04,180 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:04,180 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:04,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:04,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,182 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/ead6287b6d7f44e799e95371705bef41, entries=150, sequenceid=315, filesize=11.9 K 2023-04-25 09:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:04,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37076009[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:04,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:04,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/823d721dffa644c98aebb50247404853 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/823d721dffa644c98aebb50247404853 2023-04-25 09:55:04,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:04,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:04,185 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:04,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/823d721dffa644c98aebb50247404853, entries=150, sequenceid=315, filesize=11.9 K 2023-04-25 09:55:04,188 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:04,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 1967ec64ec5d94eb49a819b814359794 in 1246ms, sequenceid=315, compaction requested=true 2023-04-25 09:55:04,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:04,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:04,189 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:04,189 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:04,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,190 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:04,190 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:04,190 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:55:04,190 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:55:04,190 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,190 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,190 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/6bbd4aa269694b3481f1c9337fd7f789, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/65a10b22113a4bb89d4f8100d4d2801a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/498851ac49aa497ca82e08619f65cd40, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/ead6287b6d7f44e799e95371705bef41] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=48.2 K 2023-04-25 09:55:04,190 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/549e7085fed44fdea1cc365b3051c9ab, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ec8583843f2f456b89fbb6815ffbb966, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/cb56223cdf644249a2aa5e695481d6b2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d73de1c7ae2473b8d63e42d28b1ff54] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=48.2 K 2023-04-25 09:55:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,191 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6bbd4aa269694b3481f1c9337fd7f789, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682416500506 2023-04-25 09:55:04,191 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 549e7085fed44fdea1cc365b3051c9ab, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682416500506 2023-04-25 09:55:04,191 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 65a10b22113a4bb89d4f8100d4d2801a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682416501145 2023-04-25 09:55:04,191 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ec8583843f2f456b89fbb6815ffbb966, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682416501145 2023-04-25 09:55:04,191 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 498851ac49aa497ca82e08619f65cd40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682416501802 2023-04-25 09:55:04,192 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting cb56223cdf644249a2aa5e695481d6b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682416501802 2023-04-25 09:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,192 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ead6287b6d7f44e799e95371705bef41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682416502934 2023-04-25 09:55:04,192 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4d73de1c7ae2473b8d63e42d28b1ff54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682416502934 2023-04-25 09:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:04,193 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:04,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:04,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:04,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:04,193 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:04,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,198 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:04,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:04,203 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#285 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:04,203 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:04,203 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:04,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,205 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:04,216 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/4d75c55a824a40e998737df012c35d4e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d75c55a824a40e998737df012c35d4e 2023-04-25 09:55:04,220 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/19e26e37b16e4f60bfa67b9669542a32 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e26e37b16e4f60bfa67b9669542a32 2023-04-25 09:55:04,221 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into 4d75c55a824a40e998737df012c35d4e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:04,221 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:04,221 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=12, startTime=1682416504189; duration=0sec 2023-04-25 09:55:04,221 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:04,222 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:04,223 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:04,223 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:55:04,223 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,223 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/43a4d92190fc4e9d83f9d762f1d6b512, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/042435aef1e2453da5bd821efd342268, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/e49ed3eaf4ad4289b77dce9e60604003, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/823d721dffa644c98aebb50247404853] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=48.2 K 2023-04-25 09:55:04,223 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 43a4d92190fc4e9d83f9d762f1d6b512, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1682416500506 2023-04-25 09:55:04,224 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 042435aef1e2453da5bd821efd342268, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1682416501145 2023-04-25 09:55:04,224 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e49ed3eaf4ad4289b77dce9e60604003, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1682416501802 2023-04-25 09:55:04,224 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into 19e26e37b16e4f60bfa67b9669542a32(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:04,224 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:04,224 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 823d721dffa644c98aebb50247404853, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682416502934 2023-04-25 09:55:04,224 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=12, startTime=1682416504189; duration=0sec 2023-04-25 09:55:04,224 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:04,237 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:04,256 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/548d6c3d8b094c5bbf5d4200b8bea57c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/548d6c3d8b094c5bbf5d4200b8bea57c 2023-04-25 09:55:04,260 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 548d6c3d8b094c5bbf5d4200b8bea57c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:04,260 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:04,260 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=12, startTime=1682416504189; duration=0sec 2023-04-25 09:55:04,260 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:04,303 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:04,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:04,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:04,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:04,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:04,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:04,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:04,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:04,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:04,310 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=140.89 KB heapSize=390 KB 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:04,310 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:04,718 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/b9a01d8ff7284309b367778da6b41515 2023-04-25 09:55:04,728 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/1186c8c912e6463080774541a700c824 2023-04-25 09:55:04,739 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/0a77ae7021b24eb68077342180a386b9 2023-04-25 09:55:04,743 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/b9a01d8ff7284309b367778da6b41515 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b9a01d8ff7284309b367778da6b41515 2023-04-25 09:55:04,748 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b9a01d8ff7284309b367778da6b41515, entries=150, sequenceid=370, filesize=11.9 K 2023-04-25 09:55:04,749 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/1186c8c912e6463080774541a700c824 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/1186c8c912e6463080774541a700c824 2023-04-25 09:55:04,753 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/1186c8c912e6463080774541a700c824, entries=150, sequenceid=370, filesize=11.9 K 2023-04-25 09:55:04,754 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/0a77ae7021b24eb68077342180a386b9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0a77ae7021b24eb68077342180a386b9 2023-04-25 09:55:04,757 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0a77ae7021b24eb68077342180a386b9, entries=150, sequenceid=370, filesize=11.9 K 2023-04-25 09:55:04,758 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~389.95 KB/399312, currentSize=0 B/0 for 1967ec64ec5d94eb49a819b814359794 in 448ms, sequenceid=370, compaction requested=false 2023-04-25 09:55:04,758 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:04,758 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:04,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:04,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:04,758 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:04,761 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,761 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,761 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,761 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:04,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:04,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b86925[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:04,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:04,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,765 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:04,765 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:04,765 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:04,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:04,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:04,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:04,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:04,770 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:04,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:04,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:04,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:04,770 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:04,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,772 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,775 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:04,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:04,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,787 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:04,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,788 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,888 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:04,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:04,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:04,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:04,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:04,891 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:04,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:04,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:04,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,893 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,893 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,893 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,893 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:04,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:04,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:04,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:04,895 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:04,899 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:04,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a15ff67[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:04,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:04,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,903 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:04,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:04,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:04,905 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:04,908 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:04,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:04,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:04,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:04,908 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:04,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:04,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,910 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:04,912 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:04,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:04,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:04,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:04,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:04,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,915 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:04,915 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:04,915 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:04,915 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:04,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:04,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:04,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,015 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:05,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:05,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:05,018 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:05,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:05,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:05,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,021 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,021 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,021 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:05,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:05,023 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,023 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:05,023 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:05,023 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:05,023 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:05,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:05,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba4202c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:05,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:05,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:05,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:05,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:05,030 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:05,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:05,032 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:05,033 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:05,033 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:05,033 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:05,033 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:05,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,036 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,038 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:05,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:05,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,042 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:05,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,042 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:05,142 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:05,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:05,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:05,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:05,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:05,146 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:05,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:05,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:05,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,149 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:05,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:05,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:05,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:05,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:05,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:05,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:05,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:05,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272f0a79[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:05,162 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:05,162 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:05,163 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:05,163 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:05,163 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:05,163 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:05,163 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:05,163 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:05,163 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:05,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:05,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:05,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:05,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:05,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:05,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,167 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:05,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:05,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:05,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:05,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:05,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:05,170 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:05,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:05,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:05,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,176 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:05,176 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:05,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:05,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416565182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416505178, max timestamp=1682416505183Num uniques 859; 2023-04-25 09:55:05,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:05,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416505178, max timestamp=1682416505183Num uniques 859; 2023-04-25 09:55:05,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416505178, max timestamp=1682416505183Num uniques 859; 2023-04-25 09:55:05,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:05,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:05,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/589f54770d0d42d7a86148eacdd9fd50 2023-04-25 09:55:05,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416565205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416505197, max timestamp=1682416505206Num uniques 1018; 2023-04-25 09:55:05,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416505197, max timestamp=1682416505206Num uniques 1018; 2023-04-25 09:55:05,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:55:05,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416505197, max timestamp=1682416505206Num uniques 1018; 2023-04-25 09:55:05,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:55:05,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:55:05,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416565232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416565233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,276 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:05,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:05,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:05,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:05,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:05,279 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:05,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:05,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:05,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:05,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:05,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:05,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416565286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:05,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:05,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:05,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,290 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57caef8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:05,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:05,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:05,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:05,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:05,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:05,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:05,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:05,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:05,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:05,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:05,298 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:05,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,300 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:05,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:05,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,305 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:05,305 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,305 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:05,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,305 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,305 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416565310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416565335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416565337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,405 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:05,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:05,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:05,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:05,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:05,408 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:05,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:05,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:05,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:05,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:05,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:05,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:05,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:05,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:05,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:05,411 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,411 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:05,412 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:05,412 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:05,412 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:05,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@706787f7[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:05,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:05,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,417 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:05,417 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:05,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:05,418 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:05,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,418 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:05,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:05,421 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:05,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:05,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:05,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:05,421 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:05,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,425 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:05,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:05,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,428 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,428 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,428 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,429 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:05,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416565489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416565512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,529 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:05,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:05,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:05,532 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:05,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:05,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:05,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,533 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,534 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,534 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,534 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,534 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:05,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:05,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:05,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:05,535 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,536 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:05,536 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:05,536 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,537 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416565538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:05,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ac87c97[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:05,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:05,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,540 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:05,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:05,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:05,541 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,541 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:05,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416565539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,543 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:05,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:05,545 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:05,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:05,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:05,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:05,546 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:05,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:05,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:05,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,548 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:05,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:05,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:05,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,552 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:05,552 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:05,597 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/78927a4d98a64004a8ad0b9a1431a4b6 2023-04-25 09:55:05,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/159bdfeb672c4c6e95e87423bda16d86 2023-04-25 09:55:05,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/589f54770d0d42d7a86148eacdd9fd50 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/589f54770d0d42d7a86148eacdd9fd50 2023-04-25 09:55:05,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/589f54770d0d42d7a86148eacdd9fd50, entries=250, sequenceid=383, filesize=16.7 K 2023-04-25 09:55:05,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/78927a4d98a64004a8ad0b9a1431a4b6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/78927a4d98a64004a8ad0b9a1431a4b6 2023-04-25 09:55:05,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/78927a4d98a64004a8ad0b9a1431a4b6, entries=150, sequenceid=383, filesize=11.9 K 2023-04-25 09:55:05,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/159bdfeb672c4c6e95e87423bda16d86 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/159bdfeb672c4c6e95e87423bda16d86 2023-04-25 09:55:05,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/159bdfeb672c4c6e95e87423bda16d86, entries=150, sequenceid=383, filesize=11.9 K 2023-04-25 09:55:05,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1967ec64ec5d94eb49a819b814359794 in 470ms, sequenceid=383, compaction requested=true 2023-04-25 09:55:05,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:05,633 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:05,633 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:05,633 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:05,633 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:05,633 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:05,634 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:05,634 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:55:05,634 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:05,634 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,634 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:55:05,634 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d75c55a824a40e998737df012c35d4e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b9a01d8ff7284309b367778da6b41515, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/589f54770d0d42d7a86148eacdd9fd50] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=41.2 K 2023-04-25 09:55:05,634 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,635 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e26e37b16e4f60bfa67b9669542a32, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/1186c8c912e6463080774541a700c824, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/78927a4d98a64004a8ad0b9a1431a4b6] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=36.4 K 2023-04-25 09:55:05,635 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4d75c55a824a40e998737df012c35d4e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682416502934 2023-04-25 09:55:05,635 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 19e26e37b16e4f60bfa67b9669542a32, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682416502934 2023-04-25 09:55:05,635 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b9a01d8ff7284309b367778da6b41515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682416503032 2023-04-25 09:55:05,635 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1186c8c912e6463080774541a700c824, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682416503032 2023-04-25 09:55:05,635 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 589f54770d0d42d7a86148eacdd9fd50, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682416505155 2023-04-25 09:55:05,636 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 78927a4d98a64004a8ad0b9a1431a4b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682416505161 2023-04-25 09:55:05,652 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:05,652 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:05,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:05,654 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#295 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:05,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:05,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:05,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:05,660 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:05,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:05,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:05,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,664 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:05,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:05,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:05,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:05,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:05,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:05,666 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:05,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:05,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:05,669 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/f3aa62fda27b4d91a43aab161c285ad4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f3aa62fda27b4d91a43aab161c285ad4 2023-04-25 09:55:05,674 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into f3aa62fda27b4d91a43aab161c285ad4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:05,674 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:05,674 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=13, startTime=1682416505633; duration=0sec 2023-04-25 09:55:05,674 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:05,674 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:05,675 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:05,675 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:55:05,675 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:05,675 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/548d6c3d8b094c5bbf5d4200b8bea57c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0a77ae7021b24eb68077342180a386b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/159bdfeb672c4c6e95e87423bda16d86] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=36.4 K 2023-04-25 09:55:05,675 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 548d6c3d8b094c5bbf5d4200b8bea57c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1682416502934 2023-04-25 09:55:05,676 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0a77ae7021b24eb68077342180a386b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1682416503032 2023-04-25 09:55:05,676 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 159bdfeb672c4c6e95e87423bda16d86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682416505161 2023-04-25 09:55:05,681 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:05,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:05,797 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:05,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416565815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416565817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416565842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416565842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:05,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:05,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416565919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,063 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/2afcf48f9ec54b7690ffc79a4205996a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2afcf48f9ec54b7690ffc79a4205996a 2023-04-25 09:55:06,067 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into 2afcf48f9ec54b7690ffc79a4205996a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:06,067 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:06,067 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=13, startTime=1682416505633; duration=0sec 2023-04-25 09:55:06,068 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:06,074 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6d35bc0fa9384c29bd636a8a28ff09ae 2023-04-25 09:55:06,093 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/37a594427f8d469ba926e26835892309 2023-04-25 09:55:06,097 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/0f91df14928340899f055e04b3596a33 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0f91df14928340899f055e04b3596a33 2023-04-25 09:55:06,103 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 0f91df14928340899f055e04b3596a33(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:06,103 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:06,103 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=13, startTime=1682416505633; duration=0sec 2023-04-25 09:55:06,103 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:06,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:06,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416566122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:06,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416566322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:06,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416566344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:06,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416566347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:06,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416566426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,506 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bb81422a33f6467980ed9dc800d66424 2023-04-25 09:55:06,510 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6d35bc0fa9384c29bd636a8a28ff09ae as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6d35bc0fa9384c29bd636a8a28ff09ae 2023-04-25 09:55:06,514 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6d35bc0fa9384c29bd636a8a28ff09ae, entries=150, sequenceid=436, filesize=11.9 K 2023-04-25 09:55:06,515 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/37a594427f8d469ba926e26835892309 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/37a594427f8d469ba926e26835892309 2023-04-25 09:55:06,521 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/37a594427f8d469ba926e26835892309, entries=150, sequenceid=436, filesize=11.9 K 2023-04-25 09:55:06,522 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bb81422a33f6467980ed9dc800d66424 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bb81422a33f6467980ed9dc800d66424 2023-04-25 09:55:06,526 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bb81422a33f6467980ed9dc800d66424, entries=150, sequenceid=436, filesize=11.9 K 2023-04-25 09:55:06,527 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for 1967ec64ec5d94eb49a819b814359794 in 861ms, sequenceid=436, compaction requested=false 2023-04-25 09:55:06,527 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:06,527 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,527 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:06,532 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:06,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:06,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:06,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:06,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:06,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256bdd75[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:06,535 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:06,535 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,550 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:06,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:06,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:06,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:06,551 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:06,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:06,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:06,556 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:06,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:06,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:06,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:06,556 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:06,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:06,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:06,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:06,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:06,561 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:06,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:06,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:06,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:06,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:06,563 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:06,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:06,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:06,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:06,564 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,564 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,664 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:06,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:06,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:06,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:06,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:06,667 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:06,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:06,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:06,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:06,668 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,668 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:06,669 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:06,669 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:06,669 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:06,669 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:06,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:06,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:06,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:06,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:06,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:06,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:06,670 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:06,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:06,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:06,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:06,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:06,671 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:06,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:06,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:06,672 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:06,672 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:06,672 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:06,672 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:06,688 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/e59b9945922a484c92837b498d774df7 2023-04-25 09:55:06,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:06,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:06,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416506965, max timestamp=1682416506973Num uniques 859; 2023-04-25 09:55:06,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:06,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416506965, max timestamp=1682416506973Num uniques 859; 2023-04-25 09:55:06,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416506965, max timestamp=1682416506973Num uniques 859; 2023-04-25 09:55:06,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:06,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:07,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416567020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,106 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/b2490e26aef049a593a22582880d61be 2023-04-25 09:55:07,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416567123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416567325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416567327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416567353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416567356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,517 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bba0ce12b09b47b5944945b68384926d 2023-04-25 09:55:07,521 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/e59b9945922a484c92837b498d774df7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e59b9945922a484c92837b498d774df7 2023-04-25 09:55:07,527 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e59b9945922a484c92837b498d774df7, entries=150, sequenceid=449, filesize=11.9 K 2023-04-25 09:55:07,528 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/b2490e26aef049a593a22582880d61be as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/b2490e26aef049a593a22582880d61be 2023-04-25 09:55:07,531 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/b2490e26aef049a593a22582880d61be, entries=150, sequenceid=449, filesize=11.9 K 2023-04-25 09:55:07,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/bba0ce12b09b47b5944945b68384926d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bba0ce12b09b47b5944945b68384926d 2023-04-25 09:55:07,536 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bba0ce12b09b47b5944945b68384926d, entries=150, sequenceid=449, filesize=11.9 K 2023-04-25 09:55:07,537 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1967ec64ec5d94eb49a819b814359794 in 866ms, sequenceid=449, compaction requested=true 2023-04-25 09:55:07,537 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:07,537 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:07,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:07,538 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,538 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,539 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,539 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:07,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4580bc4e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:07,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:07,540 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,542 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:07,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:07,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:07,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:07,544 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:07,544 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,544 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,544 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:07,546 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:07,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:07,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:07,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:07,546 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:07,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,548 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:07,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:07,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:07,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,553 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:07,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,553 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:07,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:07,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-25 09:55:07,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:07,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:55:07,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:07,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:55:07,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:07,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:55:07,653 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:07,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416567651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:07,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:07,656 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:07,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:07,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:07,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:07,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,658 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,659 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,659 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:07,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:07,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:07,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:07,660 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,660 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:07,660 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:07,660 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:07,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:07,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:07,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@914c336[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:07,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:07,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,665 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:07,665 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:07,665 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,665 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:07,666 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:07,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:07,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:07,669 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:07,669 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:07,669 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:07,669 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:07,669 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:07,671 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,671 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,671 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,673 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:07,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:07,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,675 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:07,675 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416567754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,775 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:07,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:07,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:07,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:07,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:07,779 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:07,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:07,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:07,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:07,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,780 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,780 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:07,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:07,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:07,782 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,782 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:07,782 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:07,782 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,782 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:07,783 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:07,784 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,784 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,784 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,784 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:07,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30812a47[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:07,786 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:07,786 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:07,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:07,791 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:07,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:07,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:07,794 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:07,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:07,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:07,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:07,794 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:07,795 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,795 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,795 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,796 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:07,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:07,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,799 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:07,799 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:07,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,900 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:07,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:07,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:07,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:07,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:07,903 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:07,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:07,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:07,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:07,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:07,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:07,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:07,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:07,905 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:07,906 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,906 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:07,906 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:07,906 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:07,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:07,907 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,908 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,908 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:07,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b7ac957[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:07,909 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:07,909 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,910 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,910 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:07,910 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:07,910 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:07,910 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,910 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:07,912 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:07,912 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,912 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,912 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:07,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:07,914 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:07,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:07,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:07,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:07,914 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:07,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,916 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:07,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:07,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,918 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:07,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:07,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:07,921 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:07,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:07,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:07,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:07,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,921 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:07,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:07,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:07,922 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:07,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:07,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:07,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:07,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:07,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416567957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,022 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:08,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:08,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:08,025 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:08,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:08,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:08,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,027 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,027 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:08,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:08,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:08,029 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,029 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:08,029 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:08,029 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:08,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:08,033 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,033 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:08,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd87eba[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:08,036 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:08,036 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:08,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:08,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:08,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:08,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,038 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:08,038 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,038 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/382c582f4f3d4d39821504052f16c18c 2023-04-25 09:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:08,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:08,041 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:08,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:08,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:08,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:08,041 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:08,043 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,043 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,043 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,050 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:08,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:08,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,050 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:08,050 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:08,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,150 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:08,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:08,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:08,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:08,154 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:08,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:08,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:08,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,155 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:08,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:08,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:08,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:08,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:08,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:08,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:08,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@247a0184[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:08,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:08,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:08,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:08,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:08,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:08,163 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:08,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:08,166 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:08,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:08,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:08,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:08,166 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:08,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,170 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,174 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:08,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:08,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:08,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,174 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:08,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,175 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:08,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:08,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416568262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,275 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:08,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:08,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:08,278 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:08,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:08,278 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:08,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:08,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:08,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:08,281 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,282 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:08,282 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:08,282 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:08,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:08,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d10d45e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:08,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:08,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:08,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:08,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:08,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:08,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:08,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:08,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:08,291 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:08,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:08,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:08,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:08,291 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:08,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:08,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:08,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:08,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,298 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:08,298 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,298 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:08,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,399 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:08,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:08,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:08,402 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:08,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:08,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:08,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,403 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,404 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:08,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:08,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:08,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:08,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:08,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:08,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:08,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:08,406 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:08,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:08,407 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:08,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d96da[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:08,409 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:08,409 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,410 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:08,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:08,413 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:08,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:08,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:08,415 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:08,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:08,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:08,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:08,416 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:08,417 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,417 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:08,417 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:08,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:08,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:08,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:08,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:08,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:08,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,422 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:08,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:08,422 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:08,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/589154addff244da9b1d440d447bae5b 2023-04-25 09:55:08,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/ea31fecd6b8c470b98f724a75762dcf1 2023-04-25 09:55:08,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/382c582f4f3d4d39821504052f16c18c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/382c582f4f3d4d39821504052f16c18c 2023-04-25 09:55:08,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/382c582f4f3d4d39821504052f16c18c, entries=200, sequenceid=487, filesize=14.3 K 2023-04-25 09:55:08,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/589154addff244da9b1d440d447bae5b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/589154addff244da9b1d440d447bae5b 2023-04-25 09:55:08,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/589154addff244da9b1d440d447bae5b, entries=150, sequenceid=487, filesize=11.9 K 2023-04-25 09:55:08,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/ea31fecd6b8c470b98f724a75762dcf1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/ea31fecd6b8c470b98f724a75762dcf1 2023-04-25 09:55:08,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/ea31fecd6b8c470b98f724a75762dcf1, entries=150, sequenceid=487, filesize=11.9 K 2023-04-25 09:55:08,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 1967ec64ec5d94eb49a819b814359794 in 856ms, sequenceid=487, compaction requested=true 2023-04-25 09:55:08,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:08,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:08,489 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:08,489 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:08,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:08,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:08,490 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:08,490 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:55:08,490 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,490 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f3aa62fda27b4d91a43aab161c285ad4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6d35bc0fa9384c29bd636a8a28ff09ae, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e59b9945922a484c92837b498d774df7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/382c582f4f3d4d39821504052f16c18c] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=50.8 K 2023-04-25 09:55:08,490 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:08,490 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:55:08,490 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,491 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2afcf48f9ec54b7690ffc79a4205996a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/37a594427f8d469ba926e26835892309, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/b2490e26aef049a593a22582880d61be, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/589154addff244da9b1d440d447bae5b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=48.4 K 2023-04-25 09:55:08,491 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f3aa62fda27b4d91a43aab161c285ad4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682416505161 2023-04-25 09:55:08,491 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2afcf48f9ec54b7690ffc79a4205996a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682416505161 2023-04-25 09:55:08,491 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6d35bc0fa9384c29bd636a8a28ff09ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1682416505229 2023-04-25 09:55:08,491 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 37a594427f8d469ba926e26835892309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1682416505229 2023-04-25 09:55:08,492 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e59b9945922a484c92837b498d774df7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682416505808 2023-04-25 09:55:08,492 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b2490e26aef049a593a22582880d61be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682416505808 2023-04-25 09:55:08,492 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 382c582f4f3d4d39821504052f16c18c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416507014 2023-04-25 09:55:08,492 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 589154addff244da9b1d440d447bae5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416507016 2023-04-25 09:55:08,502 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:08,505 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#307 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:08,522 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:08,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:08,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:08,526 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/fc1ac02489d04e6f83bcb87798853fd5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/fc1ac02489d04e6f83bcb87798853fd5 2023-04-25 09:55:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:08,527 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:08,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:08,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:08,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,529 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:08,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:08,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:08,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:08,530 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/d08e6384cde746c4ad8b6c3f0b801299 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/d08e6384cde746c4ad8b6c3f0b801299 2023-04-25 09:55:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:08,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:08,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:08,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:08,531 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,531 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:08,531 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 09:55:08,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:08,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:08,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:08,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:08,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:08,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:08,533 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into fc1ac02489d04e6f83bcb87798853fd5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:08,533 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:08,533 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=12, startTime=1682416508489; duration=0sec 2023-04-25 09:55:08,533 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:08,533 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:08,535 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:08,535 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:55:08,535 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:08,535 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0f91df14928340899f055e04b3596a33, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bb81422a33f6467980ed9dc800d66424, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bba0ce12b09b47b5944945b68384926d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/ea31fecd6b8c470b98f724a75762dcf1] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=48.4 K 2023-04-25 09:55:08,536 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0f91df14928340899f055e04b3596a33, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682416505161 2023-04-25 09:55:08,536 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bb81422a33f6467980ed9dc800d66424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1682416505229 2023-04-25 09:55:08,536 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bba0ce12b09b47b5944945b68384926d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1682416505808 2023-04-25 09:55:08,536 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into d08e6384cde746c4ad8b6c3f0b801299(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:08,537 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:08,537 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=12, startTime=1682416508489; duration=0sec 2023-04-25 09:55:08,537 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ea31fecd6b8c470b98f724a75762dcf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416507016 2023-04-25 09:55:08,537 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:08,544 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:08,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:08,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:08,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416508789, max timestamp=1682416508804Num uniques 859; 2023-04-25 09:55:08,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:08,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416508789, max timestamp=1682416508804Num uniques 859; 2023-04-25 09:55:08,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:08,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416508789, max timestamp=1682416508804Num uniques 859; 2023-04-25 09:55:08,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:08,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416508842, max timestamp=1682416508852Num uniques 1018; 2023-04-25 09:55:08,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416508842, max timestamp=1682416508852Num uniques 1018; 2023-04-25 09:55:08,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682416508842, max timestamp=1682416508852Num uniques 1018; 2023-04-25 09:55:08,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 09:55:08,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 09:55:08,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-25 09:55:08,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682416508891, max timestamp=1682416508900Num uniques 1027; 2023-04-25 09:55:08,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 09:55:08,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682416508891, max timestamp=1682416508900Num uniques 1027; 2023-04-25 09:55:08,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682416508891, max timestamp=1682416508900Num uniques 1027; 2023-04-25 09:55:08,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 09:55:08,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-25 09:55:08,941 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/5ffd6c6367934cfeb7c4e629474a4372 2023-04-25 09:55:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682416508923, max timestamp=1682416508942Num uniques 1036; 2023-04-25 09:55:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682416508923, max timestamp=1682416508942Num uniques 1036; 2023-04-25 09:55:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682416508923, max timestamp=1682416508942Num uniques 1036; 2023-04-25 09:55:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 09:55:08,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 09:55:08,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-25 09:55:08,953 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/446a67eb4520435391e9facdf75f2bf2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/446a67eb4520435391e9facdf75f2bf2 2023-04-25 09:55:08,956 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 446a67eb4520435391e9facdf75f2bf2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:08,956 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:08,956 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=12, startTime=1682416508489; duration=0sec 2023-04-25 09:55:08,957 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:08,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416508969, max timestamp=1682416508993Num uniques 1045; 2023-04-25 09:55:08,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-25 09:55:08,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416508969, max timestamp=1682416508993Num uniques 1045; 2023-04-25 09:55:08,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-25 09:55:08,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416508969, max timestamp=1682416508993Num uniques 1045; 2023-04-25 09:55:08,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-25 09:55:09,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416569045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416569148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39816 deadline: 1682416569336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,339 DEBUG [Thread-1783] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:09,351 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/7befb1ba97b24bcdb887f58f5fec60d0 2023-04-25 09:55:09,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416569351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39820 deadline: 1682416569367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,370 DEBUG [Thread-1779] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:09,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39802 deadline: 1682416569371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,373 DEBUG [Thread-1787] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18243 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:09,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39844 deadline: 1682416569377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,379 DEBUG [Thread-1781] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:09,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:09,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416569653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,761 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/c63eb28e2c56480ea4507a80fab53697 2023-04-25 09:55:09,764 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/5ffd6c6367934cfeb7c4e629474a4372 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5ffd6c6367934cfeb7c4e629474a4372 2023-04-25 09:55:09,768 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5ffd6c6367934cfeb7c4e629474a4372, entries=100, sequenceid=496, filesize=9.5 K 2023-04-25 09:55:09,768 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/7befb1ba97b24bcdb887f58f5fec60d0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/7befb1ba97b24bcdb887f58f5fec60d0 2023-04-25 09:55:09,772 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/7befb1ba97b24bcdb887f58f5fec60d0, entries=100, sequenceid=496, filesize=9.5 K 2023-04-25 09:55:09,773 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/c63eb28e2c56480ea4507a80fab53697 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c63eb28e2c56480ea4507a80fab53697 2023-04-25 09:55:09,776 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c63eb28e2c56480ea4507a80fab53697, entries=100, sequenceid=496, filesize=9.5 K 2023-04-25 09:55:09,777 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 1967ec64ec5d94eb49a819b814359794 in 1246ms, sequenceid=496, compaction requested=false 2023-04-25 09:55:09,777 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:09,777 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:09,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:09,780 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,780 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:09,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:09,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d815c9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:09,782 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:09,782 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,783 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,783 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:09,783 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:09,783 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:09,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:09,784 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:09,784 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:09,784 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:09,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:09,787 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:09,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:09,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:09,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:09,787 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:09,789 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,789 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,789 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:09,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:09,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:09,790 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:09,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:09,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:09,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:09,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:09,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,797 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:09,797 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:09,797 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,798 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:09,798 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:09,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:09,898 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:09,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:09,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:09,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:09,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:09,901 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:09,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:09,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:09,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:09,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,901 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:09,902 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:09,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:09,902 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:09,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:09,903 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:09,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:09,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:09,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:09,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:09,904 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:09,904 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:09,904 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=140.89 KB heapSize=401.53 KB 2023-04-25 09:55:09,904 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:09,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:09,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:09,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:09,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:09,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:09,921 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/e0a53befafac43f29254a7f8cf38e004 2023-04-25 09:55:10,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:10,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:10,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:10,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416570168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:10,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416570271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,333 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/aeb8c05439c84cc7865585dd64a2033d 2023-04-25 09:55:10,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:10,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416570475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,744 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=605 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/34217a7d0c8b4281be5358bfcfe9bd80 2023-04-25 09:55:10,748 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/e0a53befafac43f29254a7f8cf38e004 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e0a53befafac43f29254a7f8cf38e004 2023-04-25 09:55:10,751 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e0a53befafac43f29254a7f8cf38e004, entries=150, sequenceid=605, filesize=11.9 K 2023-04-25 09:55:10,752 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/aeb8c05439c84cc7865585dd64a2033d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/aeb8c05439c84cc7865585dd64a2033d 2023-04-25 09:55:10,755 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/aeb8c05439c84cc7865585dd64a2033d, entries=150, sequenceid=605, filesize=11.9 K 2023-04-25 09:55:10,756 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/34217a7d0c8b4281be5358bfcfe9bd80 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/34217a7d0c8b4281be5358bfcfe9bd80 2023-04-25 09:55:10,759 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/34217a7d0c8b4281be5358bfcfe9bd80, entries=150, sequenceid=605, filesize=11.9 K 2023-04-25 09:55:10,760 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~401.48 KB/411120, currentSize=40.25 KB/41220 for 1967ec64ec5d94eb49a819b814359794 in 856ms, sequenceid=605, compaction requested=true 2023-04-25 09:55:10,760 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:10,760 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:10,760 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:10,760 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:10,760 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,760 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:10,760 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:10,761 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,761 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:10,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:10,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cda5b77[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:10,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:10,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,765 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:10,765 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:10,765 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:10,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:10,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:10,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:10,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:10,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:10,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:10,769 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:10,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:10,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:10,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:10,769 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:10,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:10,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:10,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:10,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:10,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:10,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:10,775 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:10,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,775 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:10,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:10,775 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:10,775 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:10,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:10,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:10,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:10,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:10,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:10,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:10,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:10,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:10,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416510813, max timestamp=1682416510830Num uniques 859; 2023-04-25 09:55:10,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416510813, max timestamp=1682416510830Num uniques 859; 2023-04-25 09:55:10,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416510813, max timestamp=1682416510830Num uniques 859; 2023-04-25 09:55:10,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:10,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:10,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:10,876 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:10,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:10,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:10,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:10,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:10,879 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:10,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:10,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:10,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:10,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,879 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,881 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:10,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:10,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:10,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416570879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:10,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:10,883 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:10,883 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:10,883 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:10,883 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:10,883 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:10,884 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,885 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:10,885 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,885 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f3da96[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:10,886 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:10,886 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,887 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,887 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:10,888 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:10,888 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:10,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:10,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,889 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:10,889 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:10,889 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:10,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:10,891 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:10,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:10,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:10,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:10,891 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:10,894 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,894 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,894 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:10,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:10,896 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,899 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:10,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:10,899 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:10,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:10,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:10,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:10,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:10,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:10,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:10,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:10,904 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:10,904 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:10,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:10,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416570983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,004 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:11,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:11,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:11,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:11,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:11,008 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:11,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:11,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:11,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:11,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:11,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:11,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:11,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:11,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:11,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:11,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:11,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:11,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:11,011 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:11,011 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. started... 2023-04-25 09:55:11,011 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. as already flushing 2023-04-25 09:55:11,011 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:11,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:11,013 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,013 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:11,013 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,013 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:11,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:11,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9413f4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:11,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:11,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:11,017 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:11,016 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:11,017 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:11,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,018 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:11,018 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:11,018 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:11,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:11,020 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:11,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:11,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:11,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:11,021 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:11,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:11,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:11,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:11,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,031 DEBUG [Thread-1796] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:60014 2023-04-25 09:55:11,031 DEBUG [Thread-1796] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:11,031 DEBUG [Thread-1798] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:60014 2023-04-25 09:55:11,031 DEBUG [Thread-1798] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:11,032 DEBUG [Thread-1794] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:60014 2023-04-25 09:55:11,032 DEBUG [Thread-1794] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:11,033 DEBUG [Thread-1792] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:60014 2023-04-25 09:55:11,033 DEBUG [Thread-1792] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:11,034 DEBUG [Thread-1790] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:60014 2023-04-25 09:55:11,034 DEBUG [Thread-1790] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:11,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:11,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:11,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:11,035 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,035 DEBUG [Thread-1789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:11,035 DEBUG [Thread-1789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:11,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:11,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:11,135 DEBUG [Thread-1789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:11,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:11,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:11,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416571186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/5949bf104225465f80ee68b13fff4632 2023-04-25 09:55:11,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:11,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416571487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:11,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/5374185608fe4c39811148edb2112074 2023-04-25 09:55:11,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:11,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39834 deadline: 1682416571989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:12,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/4938bca6ae82449185ded8f274f5aef1 2023-04-25 09:55:12,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/5949bf104225465f80ee68b13fff4632 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5949bf104225465f80ee68b13fff4632 2023-04-25 09:55:12,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5949bf104225465f80ee68b13fff4632, entries=150, sequenceid=615, filesize=11.9 K 2023-04-25 09:55:12,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/5374185608fe4c39811148edb2112074 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/5374185608fe4c39811148edb2112074 2023-04-25 09:55:12,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/5374185608fe4c39811148edb2112074, entries=150, sequenceid=615, filesize=11.9 K 2023-04-25 09:55:12,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/4938bca6ae82449185ded8f274f5aef1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/4938bca6ae82449185ded8f274f5aef1 2023-04-25 09:55:12,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/4938bca6ae82449185ded8f274f5aef1, entries=150, sequenceid=615, filesize=11.9 K 2023-04-25 09:55:12,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1967ec64ec5d94eb49a819b814359794 in 1238ms, sequenceid=615, compaction requested=true 2023-04-25 09:55:12,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:12,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:12,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:12,018 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:12,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:12,018 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:12,019 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47254 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:12,019 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47254 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:12,019 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/A is initiating minor compaction (all files) 2023-04-25 09:55:12,019 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/B is initiating minor compaction (all files) 2023-04-25 09:55:12,019 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/A in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:12,019 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/B in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:12,019 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/d08e6384cde746c4ad8b6c3f0b801299, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5ffd6c6367934cfeb7c4e629474a4372, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e0a53befafac43f29254a7f8cf38e004, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5949bf104225465f80ee68b13fff4632] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=46.1 K 2023-04-25 09:55:12,019 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/fc1ac02489d04e6f83bcb87798853fd5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/7befb1ba97b24bcdb887f58f5fec60d0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/aeb8c05439c84cc7865585dd64a2033d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/5374185608fe4c39811148edb2112074] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=46.1 K 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting d08e6384cde746c4ad8b6c3f0b801299, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416507016 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting fc1ac02489d04e6f83bcb87798853fd5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416507016 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5ffd6c6367934cfeb7c4e629474a4372, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682416507641 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7befb1ba97b24bcdb887f58f5fec60d0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682416507641 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e0a53befafac43f29254a7f8cf38e004, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682416509027 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting aeb8c05439c84cc7865585dd64a2033d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682416509027 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5949bf104225465f80ee68b13fff4632, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1682416510163 2023-04-25 09:55:12,020 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5374185608fe4c39811148edb2112074, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1682416510163 2023-04-25 09:55:12,027 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#B#compaction#318 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:12,027 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#A#compaction#319 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:12,435 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/7a2b8907a46846f28f3d0a08ccf4c091 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/7a2b8907a46846f28f3d0a08ccf4c091 2023-04-25 09:55:12,435 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/60f4c4ed4a1d4d7c9fbe5226d7c0f924 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/60f4c4ed4a1d4d7c9fbe5226d7c0f924 2023-04-25 09:55:12,439 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/B of 1967ec64ec5d94eb49a819b814359794 into 60f4c4ed4a1d4d7c9fbe5226d7c0f924(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:12,439 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/A of 1967ec64ec5d94eb49a819b814359794 into 7a2b8907a46846f28f3d0a08ccf4c091(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:12,439 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:12,439 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:12,439 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/B, priority=12, startTime=1682416512018; duration=0sec 2023-04-25 09:55:12,439 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/A, priority=12, startTime=1682416512018; duration=0sec 2023-04-25 09:55:12,439 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:12,439 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:12,439 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:12,440 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47254 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:12,440 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 1967ec64ec5d94eb49a819b814359794/C is initiating minor compaction (all files) 2023-04-25 09:55:12,440 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1967ec64ec5d94eb49a819b814359794/C in TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:12,440 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/446a67eb4520435391e9facdf75f2bf2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c63eb28e2c56480ea4507a80fab53697, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/34217a7d0c8b4281be5358bfcfe9bd80, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/4938bca6ae82449185ded8f274f5aef1] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp, totalSize=46.1 K 2023-04-25 09:55:12,440 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 446a67eb4520435391e9facdf75f2bf2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416507016 2023-04-25 09:55:12,440 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c63eb28e2c56480ea4507a80fab53697, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1682416507641 2023-04-25 09:55:12,441 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 34217a7d0c8b4281be5358bfcfe9bd80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=605, earliestPutTs=1682416509027 2023-04-25 09:55:12,441 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4938bca6ae82449185ded8f274f5aef1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1682416510163 2023-04-25 09:55:12,446 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 1967ec64ec5d94eb49a819b814359794#C#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:12,856 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/863936b9bdcb417eadf27a7914a72b04 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/863936b9bdcb417eadf27a7914a72b04 2023-04-25 09:55:12,860 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1967ec64ec5d94eb49a819b814359794/C of 1967ec64ec5d94eb49a819b814359794 into 863936b9bdcb417eadf27a7914a72b04(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:12,860 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:12,860 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794., storeName=1967ec64ec5d94eb49a819b814359794/C, priority=12, startTime=1682416512018; duration=0sec 2023-04-25 09:55:12,860 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:12,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:12,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-25 09:55:12,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:12,992 DEBUG [Thread-1785] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:60014 2023-04-25 09:55:12,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:55:12,992 DEBUG [Thread-1785] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:12,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:12,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:55:12,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:12,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:55:13,362 DEBUG [Thread-1783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:60014 2023-04-25 09:55:13,362 DEBUG [Thread-1783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:13,384 DEBUG [Thread-1779] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:60014 2023-04-25 09:55:13,384 DEBUG [Thread-1779] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:13,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6fb848b687f849e7a1282cf031d9172c 2023-04-25 09:55:13,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/d70db6878b634fc382aa0fadf185388b 2023-04-25 09:55:13,412 DEBUG [Thread-1781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:60014 2023-04-25 09:55:13,412 DEBUG [Thread-1781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:13,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/930f2a77c5a54837bb8469f8eb898a48 2023-04-25 09:55:13,824 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/6fb848b687f849e7a1282cf031d9172c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6fb848b687f849e7a1282cf031d9172c 2023-04-25 09:55:13,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6fb848b687f849e7a1282cf031d9172c, entries=150, sequenceid=656, filesize=11.9 K 2023-04-25 09:55:13,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/d70db6878b634fc382aa0fadf185388b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d70db6878b634fc382aa0fadf185388b 2023-04-25 09:55:13,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d70db6878b634fc382aa0fadf185388b, entries=150, sequenceid=656, filesize=11.9 K 2023-04-25 09:55:13,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/930f2a77c5a54837bb8469f8eb898a48 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/930f2a77c5a54837bb8469f8eb898a48 2023-04-25 09:55:13,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/930f2a77c5a54837bb8469f8eb898a48, entries=150, sequenceid=656, filesize=11.9 K 2023-04-25 09:55:13,835 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=20.13 KB/20610 for 1967ec64ec5d94eb49a819b814359794 in 843ms, sequenceid=656, compaction requested=false 2023-04-25 09:55:13,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:19,431 DEBUG [Thread-1787] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:60014 2023-04-25 09:55:19,431 DEBUG [Thread-1787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:19,431 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 09:55:19,431 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-25 09:55:19,431 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-25 09:55:19,431 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-04-25 09:55:19,431 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 268 2023-04-25 09:55:19,431 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 6096 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 6225 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 5876 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 6119 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 6228 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 09:55:19,432 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:55:19,432 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22b4eae5 to 127.0.0.1:60014 2023-04-25 09:55:19,432 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:19,433 INFO [Listener at localhost/33705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 09:55:19,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 09:55:19,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:19,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 09:55:19,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416519436"}]},"ts":"1682416519436"} 2023-04-25 09:55:19,437 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 09:55:19,440 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 09:55:19,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, UNASSIGN}] 2023-04-25 09:55:19,442 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, UNASSIGN 2023-04-25 09:55:19,443 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=1967ec64ec5d94eb49a819b814359794, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:19,443 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416519443"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416519443"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416519443"}]},"ts":"1682416519443"} 2023-04-25 09:55:19,444 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:55:19,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 09:55:19,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1967ec64ec5d94eb49a819b814359794, disabling compactions & flushes 2023-04-25 09:55:19,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. after waiting 0 ms 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:19,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1967ec64ec5d94eb49a819b814359794 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=A 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=B 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1967ec64ec5d94eb49a819b814359794, store=C 2023-04-25 09:55:19,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:19,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 09:55:20,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/b15dcc4c0dcf4529919047ee6b13a0fe 2023-04-25 09:55:20,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 09:55:20,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/2f7f6cc4710e40f98a8477e2e4a868a0 2023-04-25 09:55:20,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 09:55:20,820 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/128dc95ffdb74d47abb0917dcd945d9f 2023-04-25 09:55:20,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/A/b15dcc4c0dcf4529919047ee6b13a0fe as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b15dcc4c0dcf4529919047ee6b13a0fe 2023-04-25 09:55:20,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b15dcc4c0dcf4529919047ee6b13a0fe, entries=100, sequenceid=663, filesize=9.5 K 2023-04-25 09:55:20,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/B/2f7f6cc4710e40f98a8477e2e4a868a0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2f7f6cc4710e40f98a8477e2e4a868a0 2023-04-25 09:55:20,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2f7f6cc4710e40f98a8477e2e4a868a0, entries=100, sequenceid=663, filesize=9.5 K 2023-04-25 09:55:20,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/.tmp/C/128dc95ffdb74d47abb0917dcd945d9f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/128dc95ffdb74d47abb0917dcd945d9f 2023-04-25 09:55:20,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/128dc95ffdb74d47abb0917dcd945d9f, entries=100, sequenceid=663, filesize=9.5 K 2023-04-25 09:55:20,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1967ec64ec5d94eb49a819b814359794 in 1238ms, sequenceid=663, compaction requested=true 2023-04-25 09:55:20,856 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/a2326a322fbf4134b035b3f6237f36f4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/01015e224e7644bb8c4e9675549e096d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/40a2dc06c5bb479a8edfa44f6e7f0ee3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6c1ab76e40414f62b9cfaff2c6826e1a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ad030818889846649223a46d93e3e7e5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/dc690389f6354da9a21d81fbdfa83b28, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3c3d9f6f69824af396f53a3af9913211, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3152217a209047dfa19b24a95d84e083, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f45801e8aba449629903295d5297273a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/1e224460fc1142b598551267bb705922, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/269649c6d7484f0e8dee7f4899628efb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6f227990957d4ecf8b61d56f822db793, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/63780008ba7f48d2a16ff7d4c7fee19e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f6070f68796a45abb187e2c1962ceb87, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/8142693760fd459687e8e35154c77b17, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/0a424ff9ca09425b8a70282ccfb0f326, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5525c0aa2a56451dba26b162d3804f70, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/549e7085fed44fdea1cc365b3051c9ab, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ec8583843f2f456b89fbb6815ffbb966, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/cb56223cdf644249a2aa5e695481d6b2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d75c55a824a40e998737df012c35d4e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d73de1c7ae2473b8d63e42d28b1ff54, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b9a01d8ff7284309b367778da6b41515, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/589f54770d0d42d7a86148eacdd9fd50, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f3aa62fda27b4d91a43aab161c285ad4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6d35bc0fa9384c29bd636a8a28ff09ae, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e59b9945922a484c92837b498d774df7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/382c582f4f3d4d39821504052f16c18c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/d08e6384cde746c4ad8b6c3f0b801299, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5ffd6c6367934cfeb7c4e629474a4372, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e0a53befafac43f29254a7f8cf38e004, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5949bf104225465f80ee68b13fff4632] to archive 2023-04-25 09:55:20,857 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:20,859 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/a2326a322fbf4134b035b3f6237f36f4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/a2326a322fbf4134b035b3f6237f36f4 2023-04-25 09:55:20,860 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/01015e224e7644bb8c4e9675549e096d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/01015e224e7644bb8c4e9675549e096d 2023-04-25 09:55:20,861 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/40a2dc06c5bb479a8edfa44f6e7f0ee3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/40a2dc06c5bb479a8edfa44f6e7f0ee3 2023-04-25 09:55:20,862 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6c1ab76e40414f62b9cfaff2c6826e1a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6c1ab76e40414f62b9cfaff2c6826e1a 2023-04-25 09:55:20,864 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ad030818889846649223a46d93e3e7e5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ad030818889846649223a46d93e3e7e5 2023-04-25 09:55:20,865 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/dc690389f6354da9a21d81fbdfa83b28 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/dc690389f6354da9a21d81fbdfa83b28 2023-04-25 09:55:20,866 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3c3d9f6f69824af396f53a3af9913211 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3c3d9f6f69824af396f53a3af9913211 2023-04-25 09:55:20,867 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3152217a209047dfa19b24a95d84e083 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/3152217a209047dfa19b24a95d84e083 2023-04-25 09:55:20,868 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f45801e8aba449629903295d5297273a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f45801e8aba449629903295d5297273a 2023-04-25 09:55:20,869 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/1e224460fc1142b598551267bb705922 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/1e224460fc1142b598551267bb705922 2023-04-25 09:55:20,870 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/269649c6d7484f0e8dee7f4899628efb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/269649c6d7484f0e8dee7f4899628efb 2023-04-25 09:55:20,872 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6f227990957d4ecf8b61d56f822db793 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6f227990957d4ecf8b61d56f822db793 2023-04-25 09:55:20,873 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/63780008ba7f48d2a16ff7d4c7fee19e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/63780008ba7f48d2a16ff7d4c7fee19e 2023-04-25 09:55:20,874 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f6070f68796a45abb187e2c1962ceb87 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f6070f68796a45abb187e2c1962ceb87 2023-04-25 09:55:20,875 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/8142693760fd459687e8e35154c77b17 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/8142693760fd459687e8e35154c77b17 2023-04-25 09:55:20,876 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/0a424ff9ca09425b8a70282ccfb0f326 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/0a424ff9ca09425b8a70282ccfb0f326 2023-04-25 09:55:20,877 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5525c0aa2a56451dba26b162d3804f70 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5525c0aa2a56451dba26b162d3804f70 2023-04-25 09:55:20,878 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/549e7085fed44fdea1cc365b3051c9ab to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/549e7085fed44fdea1cc365b3051c9ab 2023-04-25 09:55:20,879 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ec8583843f2f456b89fbb6815ffbb966 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/ec8583843f2f456b89fbb6815ffbb966 2023-04-25 09:55:20,880 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/cb56223cdf644249a2aa5e695481d6b2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/cb56223cdf644249a2aa5e695481d6b2 2023-04-25 09:55:20,882 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d75c55a824a40e998737df012c35d4e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d75c55a824a40e998737df012c35d4e 2023-04-25 09:55:20,883 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d73de1c7ae2473b8d63e42d28b1ff54 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/4d73de1c7ae2473b8d63e42d28b1ff54 2023-04-25 09:55:20,884 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b9a01d8ff7284309b367778da6b41515 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b9a01d8ff7284309b367778da6b41515 2023-04-25 09:55:20,885 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/589f54770d0d42d7a86148eacdd9fd50 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/589f54770d0d42d7a86148eacdd9fd50 2023-04-25 09:55:20,886 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f3aa62fda27b4d91a43aab161c285ad4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/f3aa62fda27b4d91a43aab161c285ad4 2023-04-25 09:55:20,887 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6d35bc0fa9384c29bd636a8a28ff09ae to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6d35bc0fa9384c29bd636a8a28ff09ae 2023-04-25 09:55:20,888 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e59b9945922a484c92837b498d774df7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e59b9945922a484c92837b498d774df7 2023-04-25 09:55:20,889 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/382c582f4f3d4d39821504052f16c18c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/382c582f4f3d4d39821504052f16c18c 2023-04-25 09:55:20,890 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/d08e6384cde746c4ad8b6c3f0b801299 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/d08e6384cde746c4ad8b6c3f0b801299 2023-04-25 09:55:20,891 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5ffd6c6367934cfeb7c4e629474a4372 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5ffd6c6367934cfeb7c4e629474a4372 2023-04-25 09:55:20,893 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e0a53befafac43f29254a7f8cf38e004 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/e0a53befafac43f29254a7f8cf38e004 2023-04-25 09:55:20,894 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5949bf104225465f80ee68b13fff4632 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/5949bf104225465f80ee68b13fff4632 2023-04-25 09:55:20,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/855931fd5bfd4ec0a66663c483906b97, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/e02c1abe89254af6bf0b653c6bef46db, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/797b8c7a572440589d352406439002ba, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/efe3c16b9d214f6da00e597ab63ecfc7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e6a489bcba465aa9e2235ade650647, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/4880238e58f44a56afe49fc6ed0b1571, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/3b03156cc96e4b3eb0f01f43263b6d19, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/35701dcf66a54d73bda6bc4f05ed9ce0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d75188555cbd493593495525dd11495f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8af09b70aab84a9bb9669ce9d1e90c06, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8d9e743f023a4c86b3dbf41de4ae9fca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/00f34e2080b249468e7a07f7d21ba222, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/0bc77b1942f144c28fd1b364dad87162, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2b69a13fc0244e5aa7f627dc6906845b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/74e38d9fd37a4f818384c7cfb07ff557, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8985b0fbf3ab4584b2f84b830f2c7e37, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/6bbd4aa269694b3481f1c9337fd7f789, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/39dba5ffd72041dcbc0f83194e371fe5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/65a10b22113a4bb89d4f8100d4d2801a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/498851ac49aa497ca82e08619f65cd40, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e26e37b16e4f60bfa67b9669542a32, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/ead6287b6d7f44e799e95371705bef41, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/1186c8c912e6463080774541a700c824, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2afcf48f9ec54b7690ffc79a4205996a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/78927a4d98a64004a8ad0b9a1431a4b6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/37a594427f8d469ba926e26835892309, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/b2490e26aef049a593a22582880d61be, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/fc1ac02489d04e6f83bcb87798853fd5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/589154addff244da9b1d440d447bae5b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/7befb1ba97b24bcdb887f58f5fec60d0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/aeb8c05439c84cc7865585dd64a2033d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/5374185608fe4c39811148edb2112074] to archive 2023-04-25 09:55:20,897 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:20,899 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/855931fd5bfd4ec0a66663c483906b97 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/855931fd5bfd4ec0a66663c483906b97 2023-04-25 09:55:20,900 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/e02c1abe89254af6bf0b653c6bef46db to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/e02c1abe89254af6bf0b653c6bef46db 2023-04-25 09:55:20,901 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/797b8c7a572440589d352406439002ba to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/797b8c7a572440589d352406439002ba 2023-04-25 09:55:20,903 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/efe3c16b9d214f6da00e597ab63ecfc7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/efe3c16b9d214f6da00e597ab63ecfc7 2023-04-25 09:55:20,904 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e6a489bcba465aa9e2235ade650647 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e6a489bcba465aa9e2235ade650647 2023-04-25 09:55:20,905 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/4880238e58f44a56afe49fc6ed0b1571 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/4880238e58f44a56afe49fc6ed0b1571 2023-04-25 09:55:20,906 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/3b03156cc96e4b3eb0f01f43263b6d19 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/3b03156cc96e4b3eb0f01f43263b6d19 2023-04-25 09:55:20,907 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/35701dcf66a54d73bda6bc4f05ed9ce0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/35701dcf66a54d73bda6bc4f05ed9ce0 2023-04-25 09:55:20,908 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d75188555cbd493593495525dd11495f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d75188555cbd493593495525dd11495f 2023-04-25 09:55:20,909 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8af09b70aab84a9bb9669ce9d1e90c06 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8af09b70aab84a9bb9669ce9d1e90c06 2023-04-25 09:55:20,911 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8d9e743f023a4c86b3dbf41de4ae9fca to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8d9e743f023a4c86b3dbf41de4ae9fca 2023-04-25 09:55:20,912 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/00f34e2080b249468e7a07f7d21ba222 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/00f34e2080b249468e7a07f7d21ba222 2023-04-25 09:55:20,913 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/0bc77b1942f144c28fd1b364dad87162 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/0bc77b1942f144c28fd1b364dad87162 2023-04-25 09:55:20,914 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2b69a13fc0244e5aa7f627dc6906845b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2b69a13fc0244e5aa7f627dc6906845b 2023-04-25 09:55:20,915 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/74e38d9fd37a4f818384c7cfb07ff557 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/74e38d9fd37a4f818384c7cfb07ff557 2023-04-25 09:55:20,916 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8985b0fbf3ab4584b2f84b830f2c7e37 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/8985b0fbf3ab4584b2f84b830f2c7e37 2023-04-25 09:55:20,917 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/6bbd4aa269694b3481f1c9337fd7f789 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/6bbd4aa269694b3481f1c9337fd7f789 2023-04-25 09:55:20,918 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/39dba5ffd72041dcbc0f83194e371fe5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/39dba5ffd72041dcbc0f83194e371fe5 2023-04-25 09:55:20,920 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/65a10b22113a4bb89d4f8100d4d2801a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/65a10b22113a4bb89d4f8100d4d2801a 2023-04-25 09:55:20,921 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/498851ac49aa497ca82e08619f65cd40 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/498851ac49aa497ca82e08619f65cd40 2023-04-25 09:55:20,922 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e26e37b16e4f60bfa67b9669542a32 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/19e26e37b16e4f60bfa67b9669542a32 2023-04-25 09:55:20,923 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/ead6287b6d7f44e799e95371705bef41 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/ead6287b6d7f44e799e95371705bef41 2023-04-25 09:55:20,924 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/1186c8c912e6463080774541a700c824 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/1186c8c912e6463080774541a700c824 2023-04-25 09:55:20,925 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2afcf48f9ec54b7690ffc79a4205996a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2afcf48f9ec54b7690ffc79a4205996a 2023-04-25 09:55:20,926 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/78927a4d98a64004a8ad0b9a1431a4b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/78927a4d98a64004a8ad0b9a1431a4b6 2023-04-25 09:55:20,927 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/37a594427f8d469ba926e26835892309 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/37a594427f8d469ba926e26835892309 2023-04-25 09:55:20,928 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/b2490e26aef049a593a22582880d61be to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/b2490e26aef049a593a22582880d61be 2023-04-25 09:55:20,929 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/fc1ac02489d04e6f83bcb87798853fd5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/fc1ac02489d04e6f83bcb87798853fd5 2023-04-25 09:55:20,930 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/589154addff244da9b1d440d447bae5b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/589154addff244da9b1d440d447bae5b 2023-04-25 09:55:20,931 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/7befb1ba97b24bcdb887f58f5fec60d0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/7befb1ba97b24bcdb887f58f5fec60d0 2023-04-25 09:55:20,933 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/aeb8c05439c84cc7865585dd64a2033d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/aeb8c05439c84cc7865585dd64a2033d 2023-04-25 09:55:20,934 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/5374185608fe4c39811148edb2112074 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/5374185608fe4c39811148edb2112074 2023-04-25 09:55:20,937 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/90742755f7214efc957addd34ab26920, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dc164be2d4da409f96296db7a0e9cb22, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/876141664e1c4cbe93be45c848028207, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dfc7644feeea4f599fa9f2ba6704e4f8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/8e0853ea26234ee690e29fca920a015f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/cf5e27d84ed34ccfa60a6b602fae6a08, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6420a3b5d7cd4df9adcccf0130d76436, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbcda306cb714bf99afdbd49c16e9b37, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0ae4e7d4aa18450888d6039158ae07e3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6446cd8769ef42e6a29e71ee2838e516, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0e60e610cf2e44478b6c2ef280d52d71, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbdd15181fbf4f64a98e6cc92d203688, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/146b09fd854c4e18ae84b3a89c572a85, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/95a897008a364dc8ad7729a86146bf36, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c3f0c777fd0647369cbb69b81e9a876a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/5ba436dc9a7148eaad9105dd7e9d71b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/43a4d92190fc4e9d83f9d762f1d6b512, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/67b47c40a22941b08b0312da1ab6118b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/042435aef1e2453da5bd821efd342268, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/e49ed3eaf4ad4289b77dce9e60604003, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/548d6c3d8b094c5bbf5d4200b8bea57c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/823d721dffa644c98aebb50247404853, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0a77ae7021b24eb68077342180a386b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0f91df14928340899f055e04b3596a33, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/159bdfeb672c4c6e95e87423bda16d86, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bb81422a33f6467980ed9dc800d66424, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bba0ce12b09b47b5944945b68384926d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/446a67eb4520435391e9facdf75f2bf2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/ea31fecd6b8c470b98f724a75762dcf1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c63eb28e2c56480ea4507a80fab53697, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/34217a7d0c8b4281be5358bfcfe9bd80, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/4938bca6ae82449185ded8f274f5aef1] to archive 2023-04-25 09:55:20,938 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:20,940 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/90742755f7214efc957addd34ab26920 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/90742755f7214efc957addd34ab26920 2023-04-25 09:55:20,941 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dc164be2d4da409f96296db7a0e9cb22 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dc164be2d4da409f96296db7a0e9cb22 2023-04-25 09:55:20,942 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/876141664e1c4cbe93be45c848028207 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/876141664e1c4cbe93be45c848028207 2023-04-25 09:55:20,944 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dfc7644feeea4f599fa9f2ba6704e4f8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/dfc7644feeea4f599fa9f2ba6704e4f8 2023-04-25 09:55:20,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/8e0853ea26234ee690e29fca920a015f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/8e0853ea26234ee690e29fca920a015f 2023-04-25 09:55:20,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/cf5e27d84ed34ccfa60a6b602fae6a08 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/cf5e27d84ed34ccfa60a6b602fae6a08 2023-04-25 09:55:20,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6420a3b5d7cd4df9adcccf0130d76436 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6420a3b5d7cd4df9adcccf0130d76436 2023-04-25 09:55:20,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbcda306cb714bf99afdbd49c16e9b37 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbcda306cb714bf99afdbd49c16e9b37 2023-04-25 09:55:20,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0ae4e7d4aa18450888d6039158ae07e3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0ae4e7d4aa18450888d6039158ae07e3 2023-04-25 09:55:20,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6446cd8769ef42e6a29e71ee2838e516 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/6446cd8769ef42e6a29e71ee2838e516 2023-04-25 09:55:20,952 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0e60e610cf2e44478b6c2ef280d52d71 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0e60e610cf2e44478b6c2ef280d52d71 2023-04-25 09:55:20,953 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbdd15181fbf4f64a98e6cc92d203688 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bbdd15181fbf4f64a98e6cc92d203688 2023-04-25 09:55:20,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/146b09fd854c4e18ae84b3a89c572a85 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/146b09fd854c4e18ae84b3a89c572a85 2023-04-25 09:55:20,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/95a897008a364dc8ad7729a86146bf36 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/95a897008a364dc8ad7729a86146bf36 2023-04-25 09:55:20,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c3f0c777fd0647369cbb69b81e9a876a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c3f0c777fd0647369cbb69b81e9a876a 2023-04-25 09:55:20,956 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/5ba436dc9a7148eaad9105dd7e9d71b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/5ba436dc9a7148eaad9105dd7e9d71b1 2023-04-25 09:55:20,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/43a4d92190fc4e9d83f9d762f1d6b512 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/43a4d92190fc4e9d83f9d762f1d6b512 2023-04-25 09:55:20,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/67b47c40a22941b08b0312da1ab6118b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/67b47c40a22941b08b0312da1ab6118b 2023-04-25 09:55:20,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/042435aef1e2453da5bd821efd342268 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/042435aef1e2453da5bd821efd342268 2023-04-25 09:55:20,959 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/e49ed3eaf4ad4289b77dce9e60604003 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/e49ed3eaf4ad4289b77dce9e60604003 2023-04-25 09:55:20,959 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/548d6c3d8b094c5bbf5d4200b8bea57c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/548d6c3d8b094c5bbf5d4200b8bea57c 2023-04-25 09:55:20,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/823d721dffa644c98aebb50247404853 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/823d721dffa644c98aebb50247404853 2023-04-25 09:55:20,961 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0a77ae7021b24eb68077342180a386b9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0a77ae7021b24eb68077342180a386b9 2023-04-25 09:55:20,962 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0f91df14928340899f055e04b3596a33 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/0f91df14928340899f055e04b3596a33 2023-04-25 09:55:20,962 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/159bdfeb672c4c6e95e87423bda16d86 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/159bdfeb672c4c6e95e87423bda16d86 2023-04-25 09:55:20,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bb81422a33f6467980ed9dc800d66424 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bb81422a33f6467980ed9dc800d66424 2023-04-25 09:55:20,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bba0ce12b09b47b5944945b68384926d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/bba0ce12b09b47b5944945b68384926d 2023-04-25 09:55:20,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/446a67eb4520435391e9facdf75f2bf2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/446a67eb4520435391e9facdf75f2bf2 2023-04-25 09:55:20,965 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/ea31fecd6b8c470b98f724a75762dcf1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/ea31fecd6b8c470b98f724a75762dcf1 2023-04-25 09:55:20,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c63eb28e2c56480ea4507a80fab53697 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/c63eb28e2c56480ea4507a80fab53697 2023-04-25 09:55:20,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/34217a7d0c8b4281be5358bfcfe9bd80 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/34217a7d0c8b4281be5358bfcfe9bd80 2023-04-25 09:55:20,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/4938bca6ae82449185ded8f274f5aef1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/4938bca6ae82449185ded8f274f5aef1 2023-04-25 09:55:20,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/recovered.edits/666.seqid, newMaxSeqId=666, maxSeqId=1 2023-04-25 09:55:20,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794. 2023-04-25 09:55:20,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1967ec64ec5d94eb49a819b814359794: 2023-04-25 09:55:20,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:20,974 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=1967ec64ec5d94eb49a819b814359794, regionState=CLOSED 2023-04-25 09:55:20,974 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416520974"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416520974"}]},"ts":"1682416520974"} 2023-04-25 09:55:20,976 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-25 09:55:20,976 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 1967ec64ec5d94eb49a819b814359794, server=jenkins-hbase4.apache.org,39407,1682416417123 in 1.5310 sec 2023-04-25 09:55:20,977 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-25 09:55:20,977 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1967ec64ec5d94eb49a819b814359794, UNASSIGN in 1.5350 sec 2023-04-25 09:55:20,978 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416520977"}]},"ts":"1682416520977"} 2023-04-25 09:55:20,978 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 09:55:20,979 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 09:55:20,981 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5470 sec 2023-04-25 09:55:21,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-25 09:55:21,541 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-25 09:55:21,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 09:55:21,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,543 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-25 09:55:21,544 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,549 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:21,551 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/recovered.edits] 2023-04-25 09:55:21,553 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6fb848b687f849e7a1282cf031d9172c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/6fb848b687f849e7a1282cf031d9172c 2023-04-25 09:55:21,554 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/7a2b8907a46846f28f3d0a08ccf4c091 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/7a2b8907a46846f28f3d0a08ccf4c091 2023-04-25 09:55:21,555 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b15dcc4c0dcf4529919047ee6b13a0fe to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/A/b15dcc4c0dcf4529919047ee6b13a0fe 2023-04-25 09:55:21,557 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2f7f6cc4710e40f98a8477e2e4a868a0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/2f7f6cc4710e40f98a8477e2e4a868a0 2023-04-25 09:55:21,558 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/60f4c4ed4a1d4d7c9fbe5226d7c0f924 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/60f4c4ed4a1d4d7c9fbe5226d7c0f924 2023-04-25 09:55:21,559 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d70db6878b634fc382aa0fadf185388b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/B/d70db6878b634fc382aa0fadf185388b 2023-04-25 09:55:21,560 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/128dc95ffdb74d47abb0917dcd945d9f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/128dc95ffdb74d47abb0917dcd945d9f 2023-04-25 09:55:21,561 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/863936b9bdcb417eadf27a7914a72b04 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/863936b9bdcb417eadf27a7914a72b04 2023-04-25 09:55:21,562 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/930f2a77c5a54837bb8469f8eb898a48 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/C/930f2a77c5a54837bb8469f8eb898a48 2023-04-25 09:55:21,564 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/recovered.edits/666.seqid to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794/recovered.edits/666.seqid 2023-04-25 09:55:21,565 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/1967ec64ec5d94eb49a819b814359794 2023-04-25 09:55:21,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:55:21,566 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,567 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 09:55:21,569 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 09:55:21,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 09:55:21,570 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682416521570"}]},"ts":"9223372036854775807"} 2023-04-25 09:55:21,571 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 09:55:21,571 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1967ec64ec5d94eb49a819b814359794, NAME => 'TestAcidGuarantees,,1682416489864.1967ec64ec5d94eb49a819b814359794.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 09:55:21,571 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 09:55:21,571 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682416521571"}]},"ts":"9223372036854775807"} 2023-04-25 09:55:21,572 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 09:55:21,574 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,574 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-25 09:55:21,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-25 09:55:21,645 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-25 09:55:21,656 INFO [Listener at localhost/33705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=530 (was 518) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=318 (was 330), ProcessCount=170 (was 170), AvailableMemoryMB=4054 (was 4115) 2023-04-25 09:55:21,667 INFO [Listener at localhost/33705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=530, MaxFileDescriptor=60000, SystemLoadAverage=318, ProcessCount=170, AvailableMemoryMB=4054 2023-04-25 09:55:21,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:55:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:55:21,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:21,672 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:55:21,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-25 09:55:21,673 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:55:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 09:55:21,674 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:21,674 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 empty. 2023-04-25 09:55:21,674 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:21,674 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:55:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 09:55:21,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 09:55:22,081 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:55:22,083 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2ce6230fec57b1965ac37951c8640566, NAME => 'TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 09:55:22,489 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:55:22,489 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2ce6230fec57b1965ac37951c8640566, disabling compactions & flushes 2023-04-25 09:55:22,489 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,489 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,489 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. after waiting 0 ms 2023-04-25 09:55:22,489 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,489 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,489 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:22,491 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:55:22,492 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416522492"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416522492"}]},"ts":"1682416522492"} 2023-04-25 09:55:22,493 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:55:22,494 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:55:22,494 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416522494"}]},"ts":"1682416522494"} 2023-04-25 09:55:22,495 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 09:55:22,497 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, ASSIGN}] 2023-04-25 09:55:22,498 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, ASSIGN 2023-04-25 09:55:22,499 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:55:22,650 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:22,650 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416522650"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416522650"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416522650"}]},"ts":"1682416522650"} 2023-04-25 09:55:22,652 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:55:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 09:55:22,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2ce6230fec57b1965ac37951c8640566, NAME => 'TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:55:22,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:55:22,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,808 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,809 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A 2023-04-25 09:55:22,809 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A 2023-04-25 09:55:22,809 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:22,809 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ce6230fec57b1965ac37951c8640566 columnFamilyName A 2023-04-25 09:55:22,810 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(310): Store=2ce6230fec57b1965ac37951c8640566/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:22,810 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,811 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B 2023-04-25 09:55:22,811 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B 2023-04-25 09:55:22,811 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:22,812 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ce6230fec57b1965ac37951c8640566 columnFamilyName B 2023-04-25 09:55:22,812 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(310): Store=2ce6230fec57b1965ac37951c8640566/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:22,812 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,813 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C 2023-04-25 09:55:22,813 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C 2023-04-25 09:55:22,814 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:22,814 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ce6230fec57b1965ac37951c8640566 columnFamilyName C 2023-04-25 09:55:22,814 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(310): Store=2ce6230fec57b1965ac37951c8640566/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:22,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:55:22,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:22,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:55:22,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2ce6230fec57b1965ac37951c8640566; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59278960, jitterRate=-0.11667466163635254}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:55:22,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:22,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., pid=37, masterSystemTime=1682416522803 2023-04-25 09:55:22,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:22,823 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:22,824 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416522823"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416522823"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416522823"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416522823"}]},"ts":"1682416522823"} 2023-04-25 09:55:22,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-25 09:55:22,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 in 174 msec 2023-04-25 09:55:22,828 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-25 09:55:22,828 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, ASSIGN in 329 msec 2023-04-25 09:55:22,828 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:55:22,828 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416522828"}]},"ts":"1682416522828"} 2023-04-25 09:55:22,829 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 09:55:22,832 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:55:22,833 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1620 sec 2023-04-25 09:55:23,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-25 09:55:23,778 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-25 09:55:23,778 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:23,783 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:23,784 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:23,786 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:23,787 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:55:23,788 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:55:23,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:55:23,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:55:23,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:24,199 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 09:55:24,200 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:55:24,200 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 09:55:24,202 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 09:55:24,205 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, REOPEN/MOVE}] 2023-04-25 09:55:24,205 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, REOPEN/MOVE 2023-04-25 09:55:24,206 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,206 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416524206"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416524206"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416524206"}]},"ts":"1682416524206"} 2023-04-25 09:55:24,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:55:24,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2ce6230fec57b1965ac37951c8640566, disabling compactions & flushes 2023-04-25 09:55:24,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. after waiting 0 ms 2023-04-25 09:55:24,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 09:55:24,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:24,363 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2ce6230fec57b1965ac37951c8640566 to self. 2023-04-25 09:55:24,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,365 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=CLOSED 2023-04-25 09:55:24,365 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416524365"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416524365"}]},"ts":"1682416524365"} 2023-04-25 09:55:24,367 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-25 09:55:24,367 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 in 159 msec 2023-04-25 09:55:24,368 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=true 2023-04-25 09:55:24,518 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,518 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416524518"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416524518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416524518"}]},"ts":"1682416524518"} 2023-04-25 09:55:24,519 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:55:24,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2ce6230fec57b1965ac37951c8640566, NAME => 'TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:55:24,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:55:24,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,676 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,676 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A 2023-04-25 09:55:24,677 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A 2023-04-25 09:55:24,677 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:24,677 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ce6230fec57b1965ac37951c8640566 columnFamilyName A 2023-04-25 09:55:24,678 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(310): Store=2ce6230fec57b1965ac37951c8640566/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:24,679 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,679 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B 2023-04-25 09:55:24,679 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B 2023-04-25 09:55:24,679 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:24,680 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ce6230fec57b1965ac37951c8640566 columnFamilyName B 2023-04-25 09:55:24,680 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(310): Store=2ce6230fec57b1965ac37951c8640566/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:24,680 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,681 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C 2023-04-25 09:55:24,681 DEBUG [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C 2023-04-25 09:55:24,681 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:24,681 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ce6230fec57b1965ac37951c8640566 columnFamilyName C 2023-04-25 09:55:24,681 INFO [StoreOpener-2ce6230fec57b1965ac37951c8640566-1] regionserver.HStore(310): Store=2ce6230fec57b1965ac37951c8640566/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:24,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:55:24,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2ce6230fec57b1965ac37951c8640566; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64892810, jitterRate=-0.033021777868270874}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:55:24,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:24,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., pid=42, masterSystemTime=1682416524671 2023-04-25 09:55:24,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,688 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,688 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416524688"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416524688"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416524688"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416524688"}]},"ts":"1682416524688"} 2023-04-25 09:55:24,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-25 09:55:24,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 in 170 msec 2023-04-25 09:55:24,691 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-25 09:55:24,691 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, REOPEN/MOVE in 485 msec 2023-04-25 09:55:24,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-25 09:55:24,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-25 09:55:24,694 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-25 09:55:24,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-25 09:55:24,695 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,701 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,701 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,704 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,705 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,708 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,708 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,711 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,712 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,716 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,716 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,720 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,720 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,723 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,724 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,729 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,729 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,733 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,733 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:24,738 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:24,755 DEBUG [hconnection-0x327d89b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,756 DEBUG [hconnection-0x2f1cd898-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,757 DEBUG [hconnection-0x430077b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,757 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43010, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,759 DEBUG [hconnection-0x3cf2272-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,759 DEBUG [hconnection-0x6366f789-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,761 DEBUG [hconnection-0x10a8c857-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,761 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,762 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43026, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,770 DEBUG [hconnection-0x5294491b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,770 DEBUG [hconnection-0x354da357-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:24,771 DEBUG [hconnection-0x7be460bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,771 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,771 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,771 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,770 DEBUG [hconnection-0x346e0b18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:24,774 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,775 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,776 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,776 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:24,779 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:24,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:24,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:24,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:24,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,788 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:24,789 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:24,789 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:24,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:24,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:24,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:24,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:24,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:24,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:24,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:24,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:24,791 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,791 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:24,791 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:24,791 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:24,792 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:24,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:24,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43ea2112[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:24,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:24,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,808 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:24,808 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:24,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416584805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416584806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416584807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416584808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416524797, max timestamp=1682416524807Num uniques 859; 2023-04-25 09:55:24,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:55:24,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416524797, max timestamp=1682416524807Num uniques 859; 2023-04-25 09:55:24,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:55:24,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416524797, max timestamp=1682416524807Num uniques 859; 2023-04-25 09:55:24,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-25 09:55:24,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:24,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:24,817 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:24,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:24,819 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:24,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:24,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:24,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:24,820 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:24,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:24,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:24,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:24,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:24,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:24,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,825 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,825 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:24,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:24,835 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304253820635f595349d48a9e67965fefe6c6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253820635f595349d48a9e67965fefe6c6 2023-04-25 09:55:24,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416584836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416584910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416584911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416584911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416584913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,926 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:24,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:24,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:24,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:24,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:24,929 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:24,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:24,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:24,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:24,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,931 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:24,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:24,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:24,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:24,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:24,935 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:24,935 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,935 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:24,935 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:24,936 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:24,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:24,937 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d85dbfb[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:24,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:24,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:24,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:24,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:24,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416584937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:24,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:24,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:24,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:24,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:24,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:24,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,947 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,947 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:24,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:24,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:24,948 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:24,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:24,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:24,954 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:24,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:24,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:24,955 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:24,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:24,955 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,055 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,073 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,074 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,074 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,075 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,077 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,077 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,077 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141b83b7[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,095 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,095 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,096 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,099 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,099 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,104 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,107 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,107 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:25,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416585117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416585117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416585118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416585118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416585143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,207 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,211 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,215 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,215 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,215 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,215 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,217 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f7b3973[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,218 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,219 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,225 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,225 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,229 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,232 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,232 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,232 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,232 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,258 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/fc08ddb1931f4e54b671bc1b6c4f89d2 2023-04-25 09:55:25,332 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,335 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,337 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,337 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,337 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,339 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,339 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,339 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,339 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,341 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0b653f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,344 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,345 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,345 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,345 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,346 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,350 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,350 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,358 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,359 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,359 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,360 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416585422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416585423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416585423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416585425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416585451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,459 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,480 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,486 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,487 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:25,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,494 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,494 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,494 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,494 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,496 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ed22166[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,500 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,502 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,502 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,503 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,507 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,507 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,507 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,507 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,507 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,508 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,508 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,508 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,511 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,513 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,515 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,515 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,515 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:25,515 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,615 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,620 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,620 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,620 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,620 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,620 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,621 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,622 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,623 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,623 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,623 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:25,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,625 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,644 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,646 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261c1130[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,648 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,648 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,649 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,651 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,657 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,657 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,660 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,660 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:25,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,667 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,668 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/66081f04cca34d66b05a9e2846139e8a 2023-04-25 09:55:25,768 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,772 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,772 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,773 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,774 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,774 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:25,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,775 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,775 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,775 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,775 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,777 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a1421[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,786 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,787 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,799 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,799 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,817 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,817 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,821 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,821 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:25,922 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:25,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:25,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:25,925 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:25,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:25,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:25,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,929 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:25,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:25,931 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,931 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:25,931 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:25,931 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:25,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416585927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416585929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416585931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416585931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:25,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@369eab82[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:25,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:25,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:25,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:25,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:25,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:25,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:25,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:25,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:25,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:25,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:25,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:25,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:25,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:25,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:25,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,948 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:25,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:25,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:25,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:25,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:25,953 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:25,953 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:25,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:25,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:25,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416585960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,053 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:26,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:26,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:26,056 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:26,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:26,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:26,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:26,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:26,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:26,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:26,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:26,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:26,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:26,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49b87526[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:26,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:26,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:26,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:26,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:26,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:26,070 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:26,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:26,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:26,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:26,070 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:26,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:26,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:26,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:26,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:26,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:26,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:26,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,078 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:26,078 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:26,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,079 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,079 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,079 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/1795b635b2d74a13bb1c5eab306dd597 2023-04-25 09:55:26,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/fc08ddb1931f4e54b671bc1b6c4f89d2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/fc08ddb1931f4e54b671bc1b6c4f89d2 2023-04-25 09:55:26,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/fc08ddb1931f4e54b671bc1b6c4f89d2, entries=150, sequenceid=15, filesize=25.1 K 2023-04-25 09:55:26,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/66081f04cca34d66b05a9e2846139e8a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/66081f04cca34d66b05a9e2846139e8a 2023-04-25 09:55:26,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/66081f04cca34d66b05a9e2846139e8a, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 09:55:26,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/1795b635b2d74a13bb1c5eab306dd597 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1795b635b2d74a13bb1c5eab306dd597 2023-04-25 09:55:26,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1795b635b2d74a13bb1c5eab306dd597, entries=150, sequenceid=15, filesize=11.6 K 2023-04-25 09:55:26,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for 2ce6230fec57b1965ac37951c8640566 in 1329ms, sequenceid=15, compaction requested=false 2023-04-25 09:55:26,118 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 09:55:26,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:26,179 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:26,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:26,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:26,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:26,188 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:26,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:26,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:26,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:26,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:26,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:26,208 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:26,208 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:26,208 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=126.28 KB heapSize=362.32 KB 2023-04-25 09:55:26,208 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:26,208 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:26,208 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:26,209 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:26,209 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:26,209 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:26,253 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e48056c9607541cc94e7b1e68e0ba77c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e48056c9607541cc94e7b1e68e0ba77c 2023-04-25 09:55:26,505 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 09:55:26,678 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/7f585fc80d854bff8100052c7e261a4d 2023-04-25 09:55:26,690 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/b20f7322d3ff4674b91dc41f693e3e1d 2023-04-25 09:55:26,706 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/faecf47ecbe248348f366cbea280652e 2023-04-25 09:55:26,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/7f585fc80d854bff8100052c7e261a4d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/7f585fc80d854bff8100052c7e261a4d 2023-04-25 09:55:26,719 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/7f585fc80d854bff8100052c7e261a4d, entries=150, sequenceid=51, filesize=25.1 K 2023-04-25 09:55:26,720 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/b20f7322d3ff4674b91dc41f693e3e1d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/b20f7322d3ff4674b91dc41f693e3e1d 2023-04-25 09:55:26,728 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/b20f7322d3ff4674b91dc41f693e3e1d, entries=150, sequenceid=51, filesize=11.6 K 2023-04-25 09:55:26,729 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/faecf47ecbe248348f366cbea280652e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/faecf47ecbe248348f366cbea280652e 2023-04-25 09:55:26,736 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/faecf47ecbe248348f366cbea280652e, entries=150, sequenceid=51, filesize=11.6 K 2023-04-25 09:55:26,737 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.28 KB/129315, heapSize ~362.27 KB/370968, currentSize=0 B/0 for 2ce6230fec57b1965ac37951c8640566 in 529ms, sequenceid=51, compaction requested=false 2023-04-25 09:55:26,737 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:26,737 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:26,738 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6b78b0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:26,741 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:26,741 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,748 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:26,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:26,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:26,748 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,748 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:26,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:26,752 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:26,752 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:26,752 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:26,752 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:26,752 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:26,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:26,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,755 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,758 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:26,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:26,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:26,758 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:26,758 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,758 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,758 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:26,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:26,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,758 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:26,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:26,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:26,859 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:26,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:26,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:26,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:26,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:26,864 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:26,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:26,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:26,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,866 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:26,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:26,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:26,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:26,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:26,868 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:26,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:26,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:26,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f90ce60[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:26,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:26,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,875 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,875 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:26,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:26,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:26,880 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,881 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:26,881 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:26,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:26,884 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:26,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:26,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:26,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:26,885 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:26,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:26,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:26,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:26,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:26,887 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,890 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:26,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:26,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:26,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,891 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:26,891 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:26,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:26,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:26,991 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:26,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:26,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:26,996 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,998 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:26,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:26,998 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:26,998 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:26,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:27,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:27,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:27,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:27,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:27,008 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:27,008 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:27,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,010 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,011 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:27,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c11bcab[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:27,012 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:27,012 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,013 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:27,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:27,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:27,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,016 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:27,016 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:27,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:27,019 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:27,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:27,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:27,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:27,019 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:27,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,021 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,023 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,026 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:27,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:27,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:27,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:27,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:27,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:27,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,029 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:27,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,029 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:27,029 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:27,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:27,043 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042569315a88d81f435d9d5082032a92cfbc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042569315a88d81f435d9d5082032a92cfbc 2023-04-25 09:55:27,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416527071, max timestamp=1682416527084Num uniques 859; 2023-04-25 09:55:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416527071, max timestamp=1682416527084Num uniques 859; 2023-04-25 09:55:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416527071, max timestamp=1682416527084Num uniques 859; 2023-04-25 09:55:27,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:27,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416587104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416587111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416587112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416587113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416587113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,129 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:27,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:27,133 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:27,133 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,133 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:27,133 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:27,134 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,134 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,135 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,135 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:27,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:27,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,140 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:27,144 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,144 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:27,144 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:27,144 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:27,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,147 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168aeb4a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:27,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:27,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,150 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:27,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:27,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:27,159 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:27,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:27,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:27,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:27,159 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:27,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,168 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:27,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:27,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:27,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,169 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:27,169 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416587214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416587223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416587223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416587224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416587224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,269 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:27,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:27,279 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:27,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:27,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:27,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:27,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:27,293 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,293 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:27,293 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:27,293 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:27,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:27,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:27,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4270afd3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:27,299 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:27,299 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:27,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:27,303 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:27,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:27,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:27,308 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:27,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:27,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:27,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:27,308 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:27,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,324 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:27,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:27,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,330 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:27,330 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,330 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,330 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,330 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:27,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416587425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,430 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:27,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:27,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:27,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:27,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:27,434 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:27,434 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,434 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:27,434 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:27,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416587431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,436 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416587431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416587431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416587431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:27,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:27,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:27,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:27,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:27,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:27,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:27,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:27,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4a9a33[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:27,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:27,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:27,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:27,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:27,449 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:27,449 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:27,449 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:27,449 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:27,449 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:27,451 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,451 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:27,451 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:27,451 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,454 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/e9b26fb20d214471b9ba71c2de0e8482 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:27,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,455 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:27,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:27,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:27,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:27,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:27,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,456 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:27,456 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:27,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/695272b3a07f46a6b1359d88aefcfee1 2023-04-25 09:55:27,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/c30f8681d8c84a7a8bc1c7a6ed812458 2023-04-25 09:55:27,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/e9b26fb20d214471b9ba71c2de0e8482 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/e9b26fb20d214471b9ba71c2de0e8482 2023-04-25 09:55:27,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/e9b26fb20d214471b9ba71c2de0e8482, entries=350, sequenceid=63, filesize=52.3 K 2023-04-25 09:55:27,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/695272b3a07f46a6b1359d88aefcfee1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/695272b3a07f46a6b1359d88aefcfee1 2023-04-25 09:55:27,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/695272b3a07f46a6b1359d88aefcfee1, entries=150, sequenceid=63, filesize=11.6 K 2023-04-25 09:55:27,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/c30f8681d8c84a7a8bc1c7a6ed812458 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c30f8681d8c84a7a8bc1c7a6ed812458 2023-04-25 09:55:27,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c30f8681d8c84a7a8bc1c7a6ed812458, entries=150, sequenceid=63, filesize=11.6 K 2023-04-25 09:55:27,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ce6230fec57b1965ac37951c8640566 in 471ms, sequenceid=63, compaction requested=true 2023-04-25 09:55:27,498 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:27,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:27,498 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:27,498 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:27,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:27,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:27,499 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:27,499 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/A is initiating minor compaction (all files) 2023-04-25 09:55:27,499 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:27,499 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/A in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,499 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/B is initiating minor compaction (all files) 2023-04-25 09:55:27,499 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/fc08ddb1931f4e54b671bc1b6c4f89d2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/7f585fc80d854bff8100052c7e261a4d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/e9b26fb20d214471b9ba71c2de0e8482] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=102.5 K 2023-04-25 09:55:27,499 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/B in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,499 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/66081f04cca34d66b05a9e2846139e8a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/b20f7322d3ff4674b91dc41f693e3e1d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/695272b3a07f46a6b1359d88aefcfee1] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=34.8 K 2023-04-25 09:55:27,500 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting fc08ddb1931f4e54b671bc1b6c4f89d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682416524783 2023-04-25 09:55:27,500 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 66081f04cca34d66b05a9e2846139e8a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682416524783 2023-04-25 09:55:27,500 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 7f585fc80d854bff8100052c7e261a4d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682416524823 2023-04-25 09:55:27,500 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b20f7322d3ff4674b91dc41f693e3e1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682416524823 2023-04-25 09:55:27,500 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e9b26fb20d214471b9ba71c2de0e8482, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682416527009 2023-04-25 09:55:27,501 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 695272b3a07f46a6b1359d88aefcfee1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682416527012 2023-04-25 09:55:27,509 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#B#compaction#337 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:27,513 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#A#compaction#336 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:27,556 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:27,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:27,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:27,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:27,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:27,559 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:27,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,563 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:27,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:27,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:27,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:27,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:27,564 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-25 09:55:27,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:27,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:27,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:27,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:27,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:27,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:27,578 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425410ac0238e9b4b888ff2191d8e104f9e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425410ac0238e9b4b888ff2191d8e104f9e 2023-04-25 09:55:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:27,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:27,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416587746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416587747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416587748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416587749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416587750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416587858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416587858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416587858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416587859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:27,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416587859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:27,934 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/97b2413e2b5c47f280b0f16841b23969 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/97b2413e2b5c47f280b0f16841b23969 2023-04-25 09:55:27,939 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/B of 2ce6230fec57b1965ac37951c8640566 into 97b2413e2b5c47f280b0f16841b23969(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:27,939 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:27,939 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/B, priority=13, startTime=1682416527498; duration=0sec 2023-04-25 09:55:27,939 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:27,939 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:27,940 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:27,940 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/C is initiating minor compaction (all files) 2023-04-25 09:55:27,940 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/C in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:27,940 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1795b635b2d74a13bb1c5eab306dd597, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/faecf47ecbe248348f366cbea280652e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c30f8681d8c84a7a8bc1c7a6ed812458] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=34.8 K 2023-04-25 09:55:27,941 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1795b635b2d74a13bb1c5eab306dd597, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682416524783 2023-04-25 09:55:27,941 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting faecf47ecbe248348f366cbea280652e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682416524823 2023-04-25 09:55:27,941 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c30f8681d8c84a7a8bc1c7a6ed812458, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682416527012 2023-04-25 09:55:27,953 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#C#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:27,965 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/53990fd389f44781b6e089e857b788a1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/53990fd389f44781b6e089e857b788a1 2023-04-25 09:55:27,969 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/C of 2ce6230fec57b1965ac37951c8640566 into 53990fd389f44781b6e089e857b788a1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:27,969 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:27,969 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/C, priority=13, startTime=1682416527498; duration=0sec 2023-04-25 09:55:27,969 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:27,986 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/9cff67341999461693a3759fb7ab5d12 2023-04-25 09:55:28,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416588065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416588065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416588065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416588065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416588065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416588371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,377 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/d6f444f1feaa4eeb9e00d9f1315aff32 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/d6f444f1feaa4eeb9e00d9f1315aff32 2023-04-25 09:55:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416588380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416588380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416588380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,383 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/A of 2ce6230fec57b1965ac37951c8640566 into d6f444f1feaa4eeb9e00d9f1315aff32(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:28,383 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:28,383 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/A, priority=13, startTime=1682416527498; duration=0sec 2023-04-25 09:55:28,383 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:28,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416588381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,398 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/cf80d04b44e545c389aac45170ff8284 2023-04-25 09:55:28,866 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/1b94ee28f0c24365aafb641666faeb83 2023-04-25 09:55:28,871 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/9cff67341999461693a3759fb7ab5d12 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/9cff67341999461693a3759fb7ab5d12 2023-04-25 09:55:28,877 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/9cff67341999461693a3759fb7ab5d12, entries=150, sequenceid=98, filesize=25.1 K 2023-04-25 09:55:28,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416588877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,884 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/cf80d04b44e545c389aac45170ff8284 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cf80d04b44e545c389aac45170ff8284 2023-04-25 09:55:28,889 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cf80d04b44e545c389aac45170ff8284, entries=150, sequenceid=98, filesize=11.6 K 2023-04-25 09:55:28,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416588885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416588885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:28,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416588886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,893 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/1b94ee28f0c24365aafb641666faeb83 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b94ee28f0c24365aafb641666faeb83 2023-04-25 09:55:28,897 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b94ee28f0c24365aafb641666faeb83, entries=150, sequenceid=98, filesize=11.6 K 2023-04-25 09:55:28,898 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 2ce6230fec57b1965ac37951c8640566 in 1334ms, sequenceid=98, compaction requested=false 2023-04-25 09:55:28,898 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:28,898 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:28,899 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:28,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:28,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:55:28,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:28,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:28,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:28,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:28,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:28,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:28,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:28,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2f8915[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:28,902 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:28,902 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,905 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:28,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,906 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:28,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:28,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:28,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:28,909 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:28,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:28,912 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:28,912 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:28,912 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:28,912 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:28,912 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:28,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:28,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:28,916 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:28,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:28,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:28,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:28,920 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:28,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:28,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:28,920 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:28,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:28,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:28,921 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:28,920 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:28,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:28,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:28,932 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254b23869345a544aca51f50c72ce2761c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254b23869345a544aca51f50c72ce2761c 2023-04-25 09:55:28,954 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/bc875476a4a94bd09bb247e93492ec0d 2023-04-25 09:55:28,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/05a1108dcfdf41da99dcb1b9201a2f63 2023-04-25 09:55:28,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/555a71a3b6d2481fba5dcc0d6faba69d 2023-04-25 09:55:28,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/bc875476a4a94bd09bb247e93492ec0d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/bc875476a4a94bd09bb247e93492ec0d 2023-04-25 09:55:28,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/bc875476a4a94bd09bb247e93492ec0d, entries=200, sequenceid=112, filesize=31.9 K 2023-04-25 09:55:28,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/05a1108dcfdf41da99dcb1b9201a2f63 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/05a1108dcfdf41da99dcb1b9201a2f63 2023-04-25 09:55:29,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/05a1108dcfdf41da99dcb1b9201a2f63, entries=150, sequenceid=112, filesize=11.6 K 2023-04-25 09:55:29,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/555a71a3b6d2481fba5dcc0d6faba69d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/555a71a3b6d2481fba5dcc0d6faba69d 2023-04-25 09:55:29,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/555a71a3b6d2481fba5dcc0d6faba69d, entries=150, sequenceid=112, filesize=11.6 K 2023-04-25 09:55:29,007 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=93.93 KB/96180 for 2ce6230fec57b1965ac37951c8640566 in 106ms, sequenceid=112, compaction requested=true 2023-04-25 09:55:29,007 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:29,007 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:29,007 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:29,007 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:29,008 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:29,008 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:29,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:29,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:55:29,010 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:29,010 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:29,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:29,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:29,011 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:29,011 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:29,011 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:29,011 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/A is initiating minor compaction (all files) 2023-04-25 09:55:29,011 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/A in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,011 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/d6f444f1feaa4eeb9e00d9f1315aff32, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/9cff67341999461693a3759fb7ab5d12, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/bc875476a4a94bd09bb247e93492ec0d] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=82.3 K 2023-04-25 09:55:29,011 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:29,011 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/B is initiating minor compaction (all files) 2023-04-25 09:55:29,011 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/B in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,012 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/97b2413e2b5c47f280b0f16841b23969, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cf80d04b44e545c389aac45170ff8284, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/05a1108dcfdf41da99dcb1b9201a2f63] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=34.9 K 2023-04-25 09:55:29,012 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d6f444f1feaa4eeb9e00d9f1315aff32, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682416527012 2023-04-25 09:55:29,013 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 97b2413e2b5c47f280b0f16841b23969, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682416527012 2023-04-25 09:55:29,013 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9cff67341999461693a3759fb7ab5d12, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682416527102 2023-04-25 09:55:29,013 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting cf80d04b44e545c389aac45170ff8284, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682416527102 2023-04-25 09:55:29,014 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bc875476a4a94bd09bb247e93492ec0d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416527747 2023-04-25 09:55:29,015 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 05a1108dcfdf41da99dcb1b9201a2f63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416527747 2023-04-25 09:55:29,021 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,025 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,028 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,031 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,032 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,032 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3ef3a1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,033 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,033 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,034 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,035 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:29,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,040 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,040 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,043 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,043 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,043 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,049 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,050 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,057 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#B#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:29,061 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#A#compaction#346 average throughput is 0.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:29,089 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042521b4de78ac5048cb94c8b9ff3fb83749 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042521b4de78ac5048cb94c8b9ff3fb83749 2023-04-25 09:55:29,091 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/9adbab141e4742e28e7e6b4bcce60968 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/9adbab141e4742e28e7e6b4bcce60968 2023-04-25 09:55:29,095 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/B of 2ce6230fec57b1965ac37951c8640566 into 9adbab141e4742e28e7e6b4bcce60968(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:29,096 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:29,096 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/B, priority=13, startTime=1682416529007; duration=0sec 2023-04-25 09:55:29,096 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:29,096 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:29,097 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:29,097 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/C is initiating minor compaction (all files) 2023-04-25 09:55:29,097 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/C in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,097 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/53990fd389f44781b6e089e857b788a1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b94ee28f0c24365aafb641666faeb83, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/555a71a3b6d2481fba5dcc0d6faba69d] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=34.9 K 2023-04-25 09:55:29,097 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 53990fd389f44781b6e089e857b788a1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682416527012 2023-04-25 09:55:29,098 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1b94ee28f0c24365aafb641666faeb83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682416527102 2023-04-25 09:55:29,098 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 555a71a3b6d2481fba5dcc0d6faba69d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416527747 2023-04-25 09:55:29,105 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/5dc8b969199a44d99875c3d310d038eb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5dc8b969199a44d99875c3d310d038eb 2023-04-25 09:55:29,106 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#C#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:29,106 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/24a2b5462b8843efab2bca176848ef92 2023-04-25 09:55:29,112 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/A of 2ce6230fec57b1965ac37951c8640566 into 5dc8b969199a44d99875c3d310d038eb(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:29,112 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:29,112 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/A, priority=13, startTime=1682416529007; duration=0sec 2023-04-25 09:55:29,112 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:29,114 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/ffb1f76f43644e21bab1b3b9581cc71b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ffb1f76f43644e21bab1b3b9581cc71b 2023-04-25 09:55:29,118 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/C of 2ce6230fec57b1965ac37951c8640566 into ffb1f76f43644e21bab1b3b9581cc71b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:29,118 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:29,118 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/C, priority=13, startTime=1682416529008; duration=0sec 2023-04-25 09:55:29,118 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:29,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416589122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,150 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,153 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,153 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,155 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,157 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c77c322[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:29,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,168 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,168 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,168 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,168 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,168 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:29,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,176 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,176 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416589227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,276 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,284 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,300 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,300 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,300 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,300 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43a35b9d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,313 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,314 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,314 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,321 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,322 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:29,337 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,337 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416589430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,437 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,442 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,443 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,444 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,444 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,448 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,450 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353215dd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,453 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,453 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,454 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,455 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:29,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,458 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,459 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,468 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,469 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,469 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,472 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,472 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:29,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/cd299a2ca86f4a15be6e82b19ddbbfec 2023-04-25 09:55:29,569 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,572 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,572 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,572 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,572 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,579 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,579 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,579 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,579 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,581 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,581 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,581 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,581 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92d5b21[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,585 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,585 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,585 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,590 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,590 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,597 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,597 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,697 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,700 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,700 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,700 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,700 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,700 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,703 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,703 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,704 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,704 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,705 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,705 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,705 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,706 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf1e7f5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,709 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,710 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:29,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,713 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,713 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,716 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,719 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,719 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,720 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:29,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416589737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,820 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,828 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,831 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,831 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,831 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:29,831 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,831 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:29,832 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:29,833 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:29,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d73ca5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:29,836 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:29,836 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,838 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:29,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:29,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:29,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:29,841 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:29,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:29,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:29,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:29,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:29,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:29,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:29,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:29,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,849 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:29,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,852 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:29,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:29,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:29,853 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:29,853 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:29,853 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,853 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:29,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:29,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416589888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416589896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416589900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:29,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416589901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/9d03661a55f3408d9eed1ff6b0a768d4 2023-04-25 09:55:29,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/24a2b5462b8843efab2bca176848ef92 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/24a2b5462b8843efab2bca176848ef92 2023-04-25 09:55:29,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/24a2b5462b8843efab2bca176848ef92, entries=200, sequenceid=130, filesize=32.0 K 2023-04-25 09:55:29,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/cd299a2ca86f4a15be6e82b19ddbbfec as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cd299a2ca86f4a15be6e82b19ddbbfec 2023-04-25 09:55:29,953 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:29,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:29,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cd299a2ca86f4a15be6e82b19ddbbfec, entries=150, sequenceid=130, filesize=11.7 K 2023-04-25 09:55:29,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:29,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/9d03661a55f3408d9eed1ff6b0a768d4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/9d03661a55f3408d9eed1ff6b0a768d4 2023-04-25 09:55:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:29,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:29,960 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:29,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:29,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:29,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/9d03661a55f3408d9eed1ff6b0a768d4, entries=150, sequenceid=130, filesize=11.7 K 2023-04-25 09:55:29,964 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,964 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:29,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:29,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2ce6230fec57b1965ac37951c8640566 in 956ms, sequenceid=130, compaction requested=false 2023-04-25 09:55:29,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:29,965 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,965 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:29,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:29,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:29,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:29,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:29,966 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:29,966 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:29,966 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:29,966 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:55:29,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:29,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:29,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:29,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:29,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:29,967 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:30,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:30,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:30,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416590324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:30,388 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c06329d087094f458d3f4b5a105414ec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c06329d087094f458d3f4b5a105414ec 2023-04-25 09:55:30,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:30,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416590426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:30,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416590632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:30,793 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/0322d20fa3bf48bd8eeed0c759f1a534 2023-04-25 09:55:30,802 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/80386f1c77a2491590d86fb25a1ad831 2023-04-25 09:55:30,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:30,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416590937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,213 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/c3ff98298f344c02bacd38a076ab4d98 2023-04-25 09:55:31,217 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/0322d20fa3bf48bd8eeed0c759f1a534 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/0322d20fa3bf48bd8eeed0c759f1a534 2023-04-25 09:55:31,220 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/0322d20fa3bf48bd8eeed0c759f1a534, entries=150, sequenceid=147, filesize=25.3 K 2023-04-25 09:55:31,221 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/80386f1c77a2491590d86fb25a1ad831 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/80386f1c77a2491590d86fb25a1ad831 2023-04-25 09:55:31,224 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/80386f1c77a2491590d86fb25a1ad831, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 09:55:31,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/c3ff98298f344c02bacd38a076ab4d98 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c3ff98298f344c02bacd38a076ab4d98 2023-04-25 09:55:31,227 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c3ff98298f344c02bacd38a076ab4d98, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 09:55:31,228 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2ce6230fec57b1965ac37951c8640566 in 1262ms, sequenceid=147, compaction requested=true 2023-04-25 09:55:31,228 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:31,228 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:31,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:31,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:31,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:31,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:31,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:31,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:31,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476e41df[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:31,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:31,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,236 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:31,236 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,236 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:31,238 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:31,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,238 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:31,241 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:31,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:31,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:31,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:31,241 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:31,242 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,242 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:31,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,242 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:31,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,244 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:31,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:31,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:31,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:31,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:31,247 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:31,247 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:31,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:31,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,247 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:31,347 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:31,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:31,351 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:31,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:31,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:31,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:31,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,354 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:31,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:31,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,354 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:31,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:31,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:31,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:31,356 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:31,357 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:31,357 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:31,387 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ca12857221a84a06b01738ba25bed068 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ca12857221a84a06b01738ba25bed068 2023-04-25 09:55:31,404 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/b351dfa95c2a4499ba4cef839ed86c06 2023-04-25 09:55:31,430 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/f671aecea5f44145b42c4e060b307144 2023-04-25 09:55:31,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:31,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416591502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416591605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416591808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,843 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/3fe7494e92a34c2d8cbcc017db2131ce 2023-04-25 09:55:31,848 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/b351dfa95c2a4499ba4cef839ed86c06 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b351dfa95c2a4499ba4cef839ed86c06 2023-04-25 09:55:31,852 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b351dfa95c2a4499ba4cef839ed86c06, entries=150, sequenceid=165, filesize=25.3 K 2023-04-25 09:55:31,852 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/f671aecea5f44145b42c4e060b307144 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/f671aecea5f44145b42c4e060b307144 2023-04-25 09:55:31,856 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/f671aecea5f44145b42c4e060b307144, entries=150, sequenceid=165, filesize=11.8 K 2023-04-25 09:55:31,857 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/3fe7494e92a34c2d8cbcc017db2131ce as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/3fe7494e92a34c2d8cbcc017db2131ce 2023-04-25 09:55:31,860 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/3fe7494e92a34c2d8cbcc017db2131ce, entries=150, sequenceid=165, filesize=11.8 K 2023-04-25 09:55:31,860 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2ce6230fec57b1965ac37951c8640566 in 504ms, sequenceid=165, compaction requested=true 2023-04-25 09:55:31,860 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:31,860 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:31,860 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:31,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:31,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:31,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:31,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,865 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f9fe40[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:31,866 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:31,866 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:31,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:31,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:31,869 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:31,870 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:31,871 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:31,878 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:31,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:31,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:31,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:31,878 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:31,880 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,880 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,880 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:31,880 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,882 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:31,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:31,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:31,889 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:31,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:31,889 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:31,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:31,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:55:31,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:31,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:31,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:31,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:31,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:31,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:31,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416591936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416591939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416591940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:31,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416591943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,989 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:31,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:31,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:31,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:31,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:31,993 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:31,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:31,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:31,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:31,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:31,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:31,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:31,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:31,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:31,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:31,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:31,997 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:31,997 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:31,997 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:31,998 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:31,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,999 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10036b65[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,002 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,003 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,003 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,003 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,005 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,005 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,016 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,016 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,016 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,016 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,016 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,020 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,022 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,022 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,023 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,023 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,023 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416592046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416592046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416592049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416592051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,123 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416592117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,127 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,129 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,130 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,130 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,130 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,131 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,131 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,134 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6a5fd4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,136 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,137 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,137 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,137 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,145 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,145 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,163 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,163 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,163 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,162 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,163 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,163 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,164 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416592251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416592251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416592255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416592256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,264 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,267 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,269 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,270 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,270 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,272 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,272 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,272 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,273 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebf9fdf[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,277 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,281 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,281 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,283 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,283 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,283 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,287 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,287 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,322 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425bf5e4409eae340c8bba72fdc19f07816 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bf5e4409eae340c8bba72fdc19f07816 2023-04-25 09:55:32,387 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,391 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,391 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,394 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,394 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,397 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15df65b6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,401 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,401 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,401 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,404 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,405 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,407 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,407 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,407 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,409 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,411 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,411 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,411 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,511 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,517 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,518 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,519 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,520 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,521 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,521 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:32,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,522 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,522 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,522 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,524 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,524 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,524 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,524 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,524 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,525 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,525 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,526 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,526 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0c7c2b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,535 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,536 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,536 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,539 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,539 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,539 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,542 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,542 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,554 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,554 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,554 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,554 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,554 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416592558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416592560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416592561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416592563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:32,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43010 deadline: 1682416592627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,654 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,667 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,668 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,668 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,669 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,670 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,676 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,676 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,676 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,676 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70104c1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,680 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,680 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,681 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,684 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,684 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,692 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,692 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,736 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/31186eb402504acd8003a85a83a300bc 2023-04-25 09:55:32,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/664a955d4cbf42339260a1a21f5bcdd9 2023-04-25 09:55:32,793 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,797 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,797 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:32,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,813 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,815 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c22299[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,819 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,824 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,824 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,826 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,832 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,832 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,831 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,932 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:32,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:32,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:32,936 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:32,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:32,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:32,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,938 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:32,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:32,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:32,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:32,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:32,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:32,952 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,952 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:32,952 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:32,952 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:32,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:32,956 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bde1542[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:32,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:32,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:32,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:32,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:32,966 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:32,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:32,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:32,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:32,966 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:32,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:32,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,969 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,972 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:32,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:32,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:32,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,989 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:32,989 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:32,989 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:33,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416593066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416593067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416593067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416593068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,089 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:33,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:33,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:33,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:33,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:33,096 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:33,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:33,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:33,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:33,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:33,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:33,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:33,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:33,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:33,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:33,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:33,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:33,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:33,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:33,107 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:33,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,108 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:33,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ced136[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:33,109 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:33,109 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,111 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:33,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:33,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:33,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:33,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,113 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:33,115 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:33,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:33,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:33,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:33,115 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:33,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,117 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:33,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,118 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:33,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,119 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,121 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:33,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:33,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:33,122 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:33,122 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:33,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:33,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/7b834ca7ffb94b529bb96d8ea89eba96 2023-04-25 09:55:33,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/31186eb402504acd8003a85a83a300bc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/31186eb402504acd8003a85a83a300bc 2023-04-25 09:55:33,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/31186eb402504acd8003a85a83a300bc, entries=200, sequenceid=180, filesize=32.1 K 2023-04-25 09:55:33,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/664a955d4cbf42339260a1a21f5bcdd9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/664a955d4cbf42339260a1a21f5bcdd9 2023-04-25 09:55:33,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/664a955d4cbf42339260a1a21f5bcdd9, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 09:55:33,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/7b834ca7ffb94b529bb96d8ea89eba96 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7b834ca7ffb94b529bb96d8ea89eba96 2023-04-25 09:55:33,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7b834ca7ffb94b529bb96d8ea89eba96, entries=150, sequenceid=180, filesize=11.8 K 2023-04-25 09:55:33,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ce6230fec57b1965ac37951c8640566 in 1303ms, sequenceid=180, compaction requested=true 2023-04-25 09:55:33,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:33,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:33,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:33,209 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:55:33,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:33,209 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143366 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60136 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/A is initiating minor compaction (all files) 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/B is initiating minor compaction (all files) 2023-04-25 09:55:33,211 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/A in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:33,211 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/B in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:33,211 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5dc8b969199a44d99875c3d310d038eb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/24a2b5462b8843efab2bca176848ef92, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/0322d20fa3bf48bd8eeed0c759f1a534, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b351dfa95c2a4499ba4cef839ed86c06, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/31186eb402504acd8003a85a83a300bc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=140.0 K 2023-04-25 09:55:33,211 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/9adbab141e4742e28e7e6b4bcce60968, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cd299a2ca86f4a15be6e82b19ddbbfec, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/80386f1c77a2491590d86fb25a1ad831, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/f671aecea5f44145b42c4e060b307144, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/664a955d4cbf42339260a1a21f5bcdd9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=58.7 K 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5dc8b969199a44d99875c3d310d038eb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416527747 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9adbab141e4742e28e7e6b4bcce60968, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416527747 2023-04-25 09:55:33,211 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 24a2b5462b8843efab2bca176848ef92, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682416528983 2023-04-25 09:55:33,212 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting cd299a2ca86f4a15be6e82b19ddbbfec, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682416528983 2023-04-25 09:55:33,212 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0322d20fa3bf48bd8eeed0c759f1a534, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682416529057 2023-04-25 09:55:33,212 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 80386f1c77a2491590d86fb25a1ad831, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682416529057 2023-04-25 09:55:33,212 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b351dfa95c2a4499ba4cef839ed86c06, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682416530309 2023-04-25 09:55:33,212 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f671aecea5f44145b42c4e060b307144, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682416530309 2023-04-25 09:55:33,213 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 31186eb402504acd8003a85a83a300bc, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682416531459 2023-04-25 09:55:33,213 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 664a955d4cbf42339260a1a21f5bcdd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682416531459 2023-04-25 09:55:33,222 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:33,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:33,224 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#B#compaction#360 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:33,228 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#A#compaction#361 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:33,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:33,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:33,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:33,232 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:33,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:33,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:33,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:33,233 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,233 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:33,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:33,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:33,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:33,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:33,240 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:33,240 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:33,241 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:33,241 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:33,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:33,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:33,639 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/3f63ef9d10c74f09a8653d031a6a0257 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3f63ef9d10c74f09a8653d031a6a0257 2023-04-25 09:55:33,642 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/B of 2ce6230fec57b1965ac37951c8640566 into 3f63ef9d10c74f09a8653d031a6a0257(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:33,642 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:33,642 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/B, priority=11, startTime=1682416533209; duration=0sec 2023-04-25 09:55:33,642 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:33,642 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:55:33,644 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60136 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:55:33,644 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/C is initiating minor compaction (all files) 2023-04-25 09:55:33,644 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/C in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:33,644 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ffb1f76f43644e21bab1b3b9581cc71b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/9d03661a55f3408d9eed1ff6b0a768d4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c3ff98298f344c02bacd38a076ab4d98, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/3fe7494e92a34c2d8cbcc017db2131ce, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7b834ca7ffb94b529bb96d8ea89eba96] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=58.7 K 2023-04-25 09:55:33,644 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ffb1f76f43644e21bab1b3b9581cc71b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416527747 2023-04-25 09:55:33,645 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9d03661a55f3408d9eed1ff6b0a768d4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682416528983 2023-04-25 09:55:33,645 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c3ff98298f344c02bacd38a076ab4d98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682416529057 2023-04-25 09:55:33,646 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3fe7494e92a34c2d8cbcc017db2131ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682416530309 2023-04-25 09:55:33,646 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7b834ca7ffb94b529bb96d8ea89eba96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682416531459 2023-04-25 09:55:33,655 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#C#compaction#363 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-25 09:55:33,672 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304254031b04df97e427ea1268e3564fd1e4b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254031b04df97e427ea1268e3564fd1e4b 2023-04-25 09:55:33,679 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/a8f1a31d17f042378f6d3c5bb606ee01 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a8f1a31d17f042378f6d3c5bb606ee01 2023-04-25 09:55:33,681 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/b25cad25efaa44a78a1e2c547b1ea641 2023-04-25 09:55:33,684 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/A of 2ce6230fec57b1965ac37951c8640566 into a8f1a31d17f042378f6d3c5bb606ee01(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:33,684 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:33,684 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/A, priority=11, startTime=1682416533209; duration=0sec 2023-04-25 09:55:33,684 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:33,696 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/3d9acbd366bc43db87fd5b12d1f682fc 2023-04-25 09:55:33,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416593734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:33,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416593838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416594045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416594076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416594078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,084 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/7f1f4623740447a59f397454708f9a7f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7f1f4623740447a59f397454708f9a7f 2023-04-25 09:55:34,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416594079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416594079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,088 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/C of 2ce6230fec57b1965ac37951c8640566 into 7f1f4623740447a59f397454708f9a7f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:34,088 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:34,089 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/C, priority=11, startTime=1682416533209; duration=0sec 2023-04-25 09:55:34,089 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:34,126 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4433d0ae66804146b58b6e80efaa65ac 2023-04-25 09:55:34,130 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/b25cad25efaa44a78a1e2c547b1ea641 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b25cad25efaa44a78a1e2c547b1ea641 2023-04-25 09:55:34,134 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b25cad25efaa44a78a1e2c547b1ea641, entries=150, sequenceid=197, filesize=25.3 K 2023-04-25 09:55:34,134 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/3d9acbd366bc43db87fd5b12d1f682fc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3d9acbd366bc43db87fd5b12d1f682fc 2023-04-25 09:55:34,138 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3d9acbd366bc43db87fd5b12d1f682fc, entries=150, sequenceid=197, filesize=11.8 K 2023-04-25 09:55:34,139 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4433d0ae66804146b58b6e80efaa65ac as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4433d0ae66804146b58b6e80efaa65ac 2023-04-25 09:55:34,142 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4433d0ae66804146b58b6e80efaa65ac, entries=150, sequenceid=197, filesize=11.8 K 2023-04-25 09:55:34,142 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ce6230fec57b1965ac37951c8640566 in 901ms, sequenceid=197, compaction requested=false 2023-04-25 09:55:34,143 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:34,143 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:34,143 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:34,144 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:34,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:34,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:34,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:34,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e6e822[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:34,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:34,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,147 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:34,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:34,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:34,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:34,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:34,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:34,151 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:34,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:34,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:34,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:34,151 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:34,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:34,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:34,154 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:34,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:34,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:34,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:34,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:34,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:34,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:34,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:34,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:34,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:34,157 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,158 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:34,158 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:34,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:34,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:34,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,258 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:34,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:34,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:34,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:34,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:34,262 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:34,262 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:34,262 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:34,262 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:34,263 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,263 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:34,264 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,264 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:34,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:34,264 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,264 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:34,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:34,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:34,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:34,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:34,266 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:34,266 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:34,266 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:34,266 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:34,266 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:34,266 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:34,266 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:34,266 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:55:34,266 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:34,267 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:34,267 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:34,267 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:34,267 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:34,267 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:34,281 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ec85e42fa7bc420c825a4af9d2993517 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ec85e42fa7bc420c825a4af9d2993517 2023-04-25 09:55:34,287 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/1d59b4fd126c4d98afef637ade930a8b 2023-04-25 09:55:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:34,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:34,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416594416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416594519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:34,717 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/1a03571ede2a4ccaa0a908b1b6b9ea83 2023-04-25 09:55:34,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416594732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:35,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416595037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,139 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/6e4dec0570984e5ba01835dea751a4b1 2023-04-25 09:55:35,144 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/1d59b4fd126c4d98afef637ade930a8b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/1d59b4fd126c4d98afef637ade930a8b 2023-04-25 09:55:35,149 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/1d59b4fd126c4d98afef637ade930a8b, entries=150, sequenceid=215, filesize=25.3 K 2023-04-25 09:55:35,150 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/1a03571ede2a4ccaa0a908b1b6b9ea83 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1a03571ede2a4ccaa0a908b1b6b9ea83 2023-04-25 09:55:35,153 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1a03571ede2a4ccaa0a908b1b6b9ea83, entries=150, sequenceid=215, filesize=11.8 K 2023-04-25 09:55:35,154 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/6e4dec0570984e5ba01835dea751a4b1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/6e4dec0570984e5ba01835dea751a4b1 2023-04-25 09:55:35,157 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/6e4dec0570984e5ba01835dea751a4b1, entries=150, sequenceid=215, filesize=11.8 K 2023-04-25 09:55:35,158 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ce6230fec57b1965ac37951c8640566 in 892ms, sequenceid=215, compaction requested=true 2023-04-25 09:55:35,158 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:35,158 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:35,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:35,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:35,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,161 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:35,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:35,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da4c76d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:35,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:35,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,165 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:35,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:35,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:35,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:35,167 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:35,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:35,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:35,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:35,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:35,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:35,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:35,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:35,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:35,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:35,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:35,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:35,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:35,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:35,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:35,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:35,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:35,180 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:35,181 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:35,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:35,281 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:35,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:35,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:35,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:35,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:35,284 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:35,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:35,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:35,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:35,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:35,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:35,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:35,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,286 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:35,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:35,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:35,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:35,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:35,287 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:35,287 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:35,288 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:55:35,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:35,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:35,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:35,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:35,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:35,288 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:35,303 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251491d4c1732845a8bf878bf2c3c067b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251491d4c1732845a8bf878bf2c3c067b6 2023-04-25 09:55:35,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:35,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:35,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:35,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416595610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,710 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/696ebc63d2ef4b63b26d69aa6afb8af0 2023-04-25 09:55:35,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:35,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416595718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:35,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:35,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416595927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416596087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416596088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,091 DEBUG [Thread-2485] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:36,091 DEBUG [Thread-2487] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:36,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416596094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,098 DEBUG [Thread-2489] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:36,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416596098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,099 DEBUG [Thread-2483] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:36,122 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/933e7bd0de9c4dcfba23d4620ce721a3 2023-04-25 09:55:36,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416596232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,535 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/abf8aac53ede4c9fb363c79424a46a47 2023-04-25 09:55:36,539 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/696ebc63d2ef4b63b26d69aa6afb8af0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/696ebc63d2ef4b63b26d69aa6afb8af0 2023-04-25 09:55:36,544 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/696ebc63d2ef4b63b26d69aa6afb8af0, entries=150, sequenceid=232, filesize=25.3 K 2023-04-25 09:55:36,545 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/933e7bd0de9c4dcfba23d4620ce721a3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/933e7bd0de9c4dcfba23d4620ce721a3 2023-04-25 09:55:36,552 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/933e7bd0de9c4dcfba23d4620ce721a3, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 09:55:36,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/abf8aac53ede4c9fb363c79424a46a47 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/abf8aac53ede4c9fb363c79424a46a47 2023-04-25 09:55:36,565 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/abf8aac53ede4c9fb363c79424a46a47, entries=150, sequenceid=232, filesize=11.8 K 2023-04-25 09:55:36,566 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ce6230fec57b1965ac37951c8640566 in 1279ms, sequenceid=232, compaction requested=true 2023-04-25 09:55:36,566 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:36,566 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:36,566 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:36,566 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:36,566 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,571 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:36,571 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:36,573 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,573 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:36,573 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,573 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:36,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:36,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:36,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164bb79e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:36,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:36,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:36,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:36,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:36,580 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:36,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:36,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:36,583 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:36,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:36,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:36,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:36,583 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:36,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:36,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:36,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:36,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:36,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:36,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:36,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:36,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:36,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:36,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:36,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:36,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:36,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:36,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:36,593 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,593 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,593 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:36,593 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,593 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:36,593 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:36,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:36,693 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:36,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:36,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:36,705 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:36,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:36,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:36,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:36,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:36,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:36,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:36,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:36,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:36,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:36,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:36,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:36,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:36,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:36,709 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:36,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:36,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:36,712 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:55:36,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:36,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:36,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:36,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:36,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:36,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:36,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:36,750 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:36,756 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f8e4cf071f674eaa92ab45b44d70db39 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f8e4cf071f674eaa92ab45b44d70db39 2023-04-25 09:55:36,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416596835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:36,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:36,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416596942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:37,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416597149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,169 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=247, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/4aa2bd12623a477eb59b236bfe65b82a 2023-04-25 09:55:37,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416597456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,579 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/2e263355313c457889605d5b11be1d75 2023-04-25 09:55:37,593 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/d3c63184a97842c7872117fc0c591edf 2023-04-25 09:55:37,598 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/4aa2bd12623a477eb59b236bfe65b82a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/4aa2bd12623a477eb59b236bfe65b82a 2023-04-25 09:55:37,601 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/4aa2bd12623a477eb59b236bfe65b82a, entries=150, sequenceid=247, filesize=25.3 K 2023-04-25 09:55:37,602 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/2e263355313c457889605d5b11be1d75 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/2e263355313c457889605d5b11be1d75 2023-04-25 09:55:37,606 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/2e263355313c457889605d5b11be1d75, entries=150, sequenceid=247, filesize=11.8 K 2023-04-25 09:55:37,606 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/d3c63184a97842c7872117fc0c591edf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d3c63184a97842c7872117fc0c591edf 2023-04-25 09:55:37,609 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d3c63184a97842c7872117fc0c591edf, entries=150, sequenceid=247, filesize=11.8 K 2023-04-25 09:55:37,610 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ce6230fec57b1965ac37951c8640566 in 898ms, sequenceid=247, compaction requested=true 2023-04-25 09:55:37,610 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:37,610 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:37,610 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:37,610 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:37,610 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,610 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:37,610 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:37,612 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:37,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:37,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76ec8300[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:37,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:37,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,616 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:37,621 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,621 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:37,623 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,623 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:37,623 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:37,624 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:37,626 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:37,626 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:37,626 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:37,626 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:37,626 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:37,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:37,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,633 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:37,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:37,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:37,637 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:37,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:37,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:37,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:37,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:37,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:37,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:37,638 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:37,638 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:37,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:37,738 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:37,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:37,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:37,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:37,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:37,746 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:37,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:37,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:37,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:37,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,747 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:37,748 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:37,748 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:37,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:37,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:37,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:37,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:37,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:37,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:37,750 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:37,750 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:37,750 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:37,751 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:37,751 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:37,751 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:37,751 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:37,782 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a444683f945f4e13af9f181f5f0c52ec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a444683f945f4e13af9f181f5f0c52ec 2023-04-25 09:55:37,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:37,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:38,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:38,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416598075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,189 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/ec23cfce2442407285a3e1f1ca8653cb 2023-04-25 09:55:38,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:38,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416598184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,199 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/328aa63be66e4aa5a8dd90eef807bbcf 2023-04-25 09:55:38,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:38,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416598398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,609 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/c7921b0c945349d4a878ffe66c366f96 2023-04-25 09:55:38,615 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/ec23cfce2442407285a3e1f1ca8653cb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/ec23cfce2442407285a3e1f1ca8653cb 2023-04-25 09:55:38,627 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/ec23cfce2442407285a3e1f1ca8653cb, entries=150, sequenceid=264, filesize=25.4 K 2023-04-25 09:55:38,628 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/328aa63be66e4aa5a8dd90eef807bbcf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/328aa63be66e4aa5a8dd90eef807bbcf 2023-04-25 09:55:38,633 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/328aa63be66e4aa5a8dd90eef807bbcf, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 09:55:38,636 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/c7921b0c945349d4a878ffe66c366f96 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c7921b0c945349d4a878ffe66c366f96 2023-04-25 09:55:38,645 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c7921b0c945349d4a878ffe66c366f96, entries=150, sequenceid=264, filesize=11.9 K 2023-04-25 09:55:38,646 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ce6230fec57b1965ac37951c8640566 in 895ms, sequenceid=264, compaction requested=true 2023-04-25 09:55:38,646 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:38,646 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:38,646 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,648 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:38,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:38,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e5d207[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:38,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:38,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,651 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:38,651 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:38,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:38,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:38,653 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:38,652 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:38,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:38,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:38,663 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:38,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:38,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:38,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:38,663 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:38,665 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,665 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:38,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,666 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:38,666 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,668 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:38,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,673 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:38,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:38,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:38,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:38,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:38,675 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:38,675 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:38,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:38,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 09:55:38,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:38,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:38,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:38,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:38,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:38,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:38,775 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:38,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:38,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:38,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:38,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:38,810 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:38,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:38,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:38,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:38,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:38,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:38,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:38,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:38,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:38,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:38,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:38,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:38,828 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:38,828 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:38,828 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:38,828 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:38,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:38,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:38,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:38,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:38,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:38,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7351f9d9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:38,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:38,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,842 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:38,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:38,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:38,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:38,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:38,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:38,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:38,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:38,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:38,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:38,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:38,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:38,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,847 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:38,848 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,849 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,852 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:38,852 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:38,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:38,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:38,853 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:38,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:38,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:38,853 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:38,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:38,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:38,855 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:38,855 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:38,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:38,907 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.23076923076923078, tune throughput to 61.54 MB/second 2023-04-25 09:55:38,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:38,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416598893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:38,955 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:38,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:38,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:39,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:39,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:39,008 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:39,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:39,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:39,013 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,013 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,014 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:39,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:39,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:39,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416599013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:39,029 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/fc08ddb1931f4e54b671bc1b6c4f89d2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/7f585fc80d854bff8100052c7e261a4d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/e9b26fb20d214471b9ba71c2de0e8482, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/d6f444f1feaa4eeb9e00d9f1315aff32, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/9cff67341999461693a3759fb7ab5d12, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/bc875476a4a94bd09bb247e93492ec0d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5dc8b969199a44d99875c3d310d038eb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/24a2b5462b8843efab2bca176848ef92, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/0322d20fa3bf48bd8eeed0c759f1a534, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b351dfa95c2a4499ba4cef839ed86c06, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/31186eb402504acd8003a85a83a300bc] to archive 2023-04-25 09:55:39,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:39,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:39,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:39,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:39,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:39,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:39,030 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:39,030 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:39,032 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,032 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/fc08ddb1931f4e54b671bc1b6c4f89d2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/fc08ddb1931f4e54b671bc1b6c4f89d2 2023-04-25 09:55:39,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,033 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:39,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,034 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/7f585fc80d854bff8100052c7e261a4d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/7f585fc80d854bff8100052c7e261a4d 2023-04-25 09:55:39,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:39,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386d576a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:39,035 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:39,035 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,036 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/e9b26fb20d214471b9ba71c2de0e8482 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/e9b26fb20d214471b9ba71c2de0e8482 2023-04-25 09:55:39,036 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:39,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:39,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:39,038 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/d6f444f1feaa4eeb9e00d9f1315aff32 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/d6f444f1feaa4eeb9e00d9f1315aff32 2023-04-25 09:55:39,039 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:39,039 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,039 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:39,039 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,040 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/9cff67341999461693a3759fb7ab5d12 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/9cff67341999461693a3759fb7ab5d12 2023-04-25 09:55:39,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,042 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/bc875476a4a94bd09bb247e93492ec0d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/bc875476a4a94bd09bb247e93492ec0d 2023-04-25 09:55:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,043 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5dc8b969199a44d99875c3d310d038eb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5dc8b969199a44d99875c3d310d038eb 2023-04-25 09:55:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:39,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:39,044 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:39,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:39,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:39,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:39,044 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:39,046 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/24a2b5462b8843efab2bca176848ef92 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/24a2b5462b8843efab2bca176848ef92 2023-04-25 09:55:39,047 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/0322d20fa3bf48bd8eeed0c759f1a534 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/0322d20fa3bf48bd8eeed0c759f1a534 2023-04-25 09:55:39,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,049 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,051 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b351dfa95c2a4499ba4cef839ed86c06 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b351dfa95c2a4499ba4cef839ed86c06 2023-04-25 09:55:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,052 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,053 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/31186eb402504acd8003a85a83a300bc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/31186eb402504acd8003a85a83a300bc 2023-04-25 09:55:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,070 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,070 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,071 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:39,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:39,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:39,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,073 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:39,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,077 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,077 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:39,110 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/66081f04cca34d66b05a9e2846139e8a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/b20f7322d3ff4674b91dc41f693e3e1d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/97b2413e2b5c47f280b0f16841b23969, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/695272b3a07f46a6b1359d88aefcfee1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cf80d04b44e545c389aac45170ff8284, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/9adbab141e4742e28e7e6b4bcce60968, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/05a1108dcfdf41da99dcb1b9201a2f63, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cd299a2ca86f4a15be6e82b19ddbbfec, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/80386f1c77a2491590d86fb25a1ad831, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/f671aecea5f44145b42c4e060b307144, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/664a955d4cbf42339260a1a21f5bcdd9] to archive 2023-04-25 09:55:39,112 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:39,114 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/66081f04cca34d66b05a9e2846139e8a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/66081f04cca34d66b05a9e2846139e8a 2023-04-25 09:55:39,115 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/b20f7322d3ff4674b91dc41f693e3e1d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/b20f7322d3ff4674b91dc41f693e3e1d 2023-04-25 09:55:39,117 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/97b2413e2b5c47f280b0f16841b23969 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/97b2413e2b5c47f280b0f16841b23969 2023-04-25 09:55:39,118 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/695272b3a07f46a6b1359d88aefcfee1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/695272b3a07f46a6b1359d88aefcfee1 2023-04-25 09:55:39,120 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cf80d04b44e545c389aac45170ff8284 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cf80d04b44e545c389aac45170ff8284 2023-04-25 09:55:39,121 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/9adbab141e4742e28e7e6b4bcce60968 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/9adbab141e4742e28e7e6b4bcce60968 2023-04-25 09:55:39,123 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/05a1108dcfdf41da99dcb1b9201a2f63 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/05a1108dcfdf41da99dcb1b9201a2f63 2023-04-25 09:55:39,124 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cd299a2ca86f4a15be6e82b19ddbbfec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/cd299a2ca86f4a15be6e82b19ddbbfec 2023-04-25 09:55:39,128 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/80386f1c77a2491590d86fb25a1ad831 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/80386f1c77a2491590d86fb25a1ad831 2023-04-25 09:55:39,130 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/f671aecea5f44145b42c4e060b307144 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/f671aecea5f44145b42c4e060b307144 2023-04-25 09:55:39,132 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/664a955d4cbf42339260a1a21f5bcdd9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/664a955d4cbf42339260a1a21f5bcdd9 2023-04-25 09:55:39,136 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1795b635b2d74a13bb1c5eab306dd597, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/faecf47ecbe248348f366cbea280652e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/53990fd389f44781b6e089e857b788a1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c30f8681d8c84a7a8bc1c7a6ed812458, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b94ee28f0c24365aafb641666faeb83, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ffb1f76f43644e21bab1b3b9581cc71b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/555a71a3b6d2481fba5dcc0d6faba69d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/9d03661a55f3408d9eed1ff6b0a768d4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c3ff98298f344c02bacd38a076ab4d98, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/3fe7494e92a34c2d8cbcc017db2131ce, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7b834ca7ffb94b529bb96d8ea89eba96] to archive 2023-04-25 09:55:39,136 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:39,138 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1795b635b2d74a13bb1c5eab306dd597 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1795b635b2d74a13bb1c5eab306dd597 2023-04-25 09:55:39,140 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/faecf47ecbe248348f366cbea280652e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/faecf47ecbe248348f366cbea280652e 2023-04-25 09:55:39,141 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/53990fd389f44781b6e089e857b788a1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/53990fd389f44781b6e089e857b788a1 2023-04-25 09:55:39,142 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c30f8681d8c84a7a8bc1c7a6ed812458 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c30f8681d8c84a7a8bc1c7a6ed812458 2023-04-25 09:55:39,143 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b94ee28f0c24365aafb641666faeb83 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b94ee28f0c24365aafb641666faeb83 2023-04-25 09:55:39,145 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ffb1f76f43644e21bab1b3b9581cc71b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ffb1f76f43644e21bab1b3b9581cc71b 2023-04-25 09:55:39,147 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/555a71a3b6d2481fba5dcc0d6faba69d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/555a71a3b6d2481fba5dcc0d6faba69d 2023-04-25 09:55:39,165 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/9d03661a55f3408d9eed1ff6b0a768d4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/9d03661a55f3408d9eed1ff6b0a768d4 2023-04-25 09:55:39,167 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c3ff98298f344c02bacd38a076ab4d98 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c3ff98298f344c02bacd38a076ab4d98 2023-04-25 09:55:39,168 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/3fe7494e92a34c2d8cbcc017db2131ce to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/3fe7494e92a34c2d8cbcc017db2131ce 2023-04-25 09:55:39,171 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7b834ca7ffb94b529bb96d8ea89eba96 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7b834ca7ffb94b529bb96d8ea89eba96 2023-04-25 09:55:39,176 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251348f6cd4a25493abf5f2ebd88c80d90 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251348f6cd4a25493abf5f2ebd88c80d90 2023-04-25 09:55:39,179 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:39,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:39,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:39,205 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:39,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:39,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:39,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,209 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/979a9c4af53a4b5995e0412455e65861 2023-04-25 09:55:39,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:39,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:39,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:39,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:39,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:39,226 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,226 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:39,228 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:39,228 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:39,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:39,231 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a8b27e4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:39,234 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:39,234 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,235 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:39,235 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:39,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:39,235 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:39,236 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:39,239 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:39,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:39,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:39,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:39,239 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:39,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,241 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:39,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416599228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,242 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,246 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:39,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:39,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:39,247 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:39,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,247 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:39,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/bbb3c5f0892843f2a3772771496561d7 2023-04-25 09:55:39,347 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:39,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:39,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:39,363 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:39,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:39,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:39,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,365 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:39,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:39,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:39,380 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:39,380 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,380 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:39,384 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,384 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:39,386 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:39,386 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:39,387 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,388 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,391 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,391 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:39,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c08bd58[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:39,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:39,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,394 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,394 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:39,394 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:39,394 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:39,395 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,395 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:39,401 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:39,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:39,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:39,410 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:39,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:39,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:39,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:39,410 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:39,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,412 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,414 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:39,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:39,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,421 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,421 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:39,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,421 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:39,521 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:39,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:39,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:39,528 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,530 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,530 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:39,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:39,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:39,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:39,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:39,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:39,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:39,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:39,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:39,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:39,547 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,547 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,549 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20eb0831[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:39,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:39,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,551 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:39,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:39,551 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:39,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:39,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:39,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416599542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:39,557 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:39,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:39,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:39,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:39,557 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:39,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,560 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,561 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,564 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:39,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,568 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:39,568 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:39,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:39,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,669 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:39,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:39,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:39,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:39,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:39,674 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:39,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:39,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:39,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,676 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:39,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:39,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:39,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:39,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:39,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:39,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:39,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:39,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,694 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:39,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:39,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726fcd41[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:39,697 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:39,698 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:39,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:39,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:39,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:39,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:39,704 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:39,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:39,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:39,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:39,705 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:39,706 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,706 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,706 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/d5682afcf1d94470bd6c8e3d4b6a964f 2023-04-25 09:55:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:39,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/979a9c4af53a4b5995e0412455e65861 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/979a9c4af53a4b5995e0412455e65861 2023-04-25 09:55:39,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:39,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,715 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:39,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:39,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,715 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:39,715 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:39,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:39,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/979a9c4af53a4b5995e0412455e65861, entries=200, sequenceid=280, filesize=32.3 K 2023-04-25 09:55:39,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/bbb3c5f0892843f2a3772771496561d7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/bbb3c5f0892843f2a3772771496561d7 2023-04-25 09:55:39,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/bbb3c5f0892843f2a3772771496561d7, entries=150, sequenceid=280, filesize=11.9 K 2023-04-25 09:55:39,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/d5682afcf1d94470bd6c8e3d4b6a964f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d5682afcf1d94470bd6c8e3d4b6a964f 2023-04-25 09:55:39,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d5682afcf1d94470bd6c8e3d4b6a964f, entries=150, sequenceid=280, filesize=11.9 K 2023-04-25 09:55:39,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2ce6230fec57b1965ac37951c8640566 in 1027ms, sequenceid=280, compaction requested=true 2023-04-25 09:55:39,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:39,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:39,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:39,744 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 09:55:39,745 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 09:55:39,745 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:55:39,747 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188988 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 09:55:39,747 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/A is initiating minor compaction (all files) 2023-04-25 09:55:39,747 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/A in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,747 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a8f1a31d17f042378f6d3c5bb606ee01, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b25cad25efaa44a78a1e2c547b1ea641, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/1d59b4fd126c4d98afef637ade930a8b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/696ebc63d2ef4b63b26d69aa6afb8af0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/4aa2bd12623a477eb59b236bfe65b82a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/ec23cfce2442407285a3e1f1ca8653cb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/979a9c4af53a4b5995e0412455e65861] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=184.6 K 2023-04-25 09:55:39,748 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84928 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 09:55:39,748 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/B is initiating minor compaction (all files) 2023-04-25 09:55:39,748 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/B in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,748 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3f63ef9d10c74f09a8653d031a6a0257, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3d9acbd366bc43db87fd5b12d1f682fc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1a03571ede2a4ccaa0a908b1b6b9ea83, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/933e7bd0de9c4dcfba23d4620ce721a3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/2e263355313c457889605d5b11be1d75, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/328aa63be66e4aa5a8dd90eef807bbcf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/bbb3c5f0892843f2a3772771496561d7] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=82.9 K 2023-04-25 09:55:39,749 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3f63ef9d10c74f09a8653d031a6a0257, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682416531459 2023-04-25 09:55:39,749 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3d9acbd366bc43db87fd5b12d1f682fc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682416531937 2023-04-25 09:55:39,749 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a8f1a31d17f042378f6d3c5bb606ee01, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682416531459 2023-04-25 09:55:39,750 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1a03571ede2a4ccaa0a908b1b6b9ea83, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416533653 2023-04-25 09:55:39,750 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 933e7bd0de9c4dcfba23d4620ce721a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682416534406 2023-04-25 09:55:39,750 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2e263355313c457889605d5b11be1d75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1682416535585 2023-04-25 09:55:39,751 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b25cad25efaa44a78a1e2c547b1ea641, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682416531937 2023-04-25 09:55:39,751 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1d59b4fd126c4d98afef637ade930a8b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416533653 2023-04-25 09:55:39,751 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 696ebc63d2ef4b63b26d69aa6afb8af0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682416534406 2023-04-25 09:55:39,752 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 328aa63be66e4aa5a8dd90eef807bbcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682416536817 2023-04-25 09:55:39,752 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4aa2bd12623a477eb59b236bfe65b82a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1682416535585 2023-04-25 09:55:39,752 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bbb3c5f0892843f2a3772771496561d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682416538039 2023-04-25 09:55:39,753 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ec23cfce2442407285a3e1f1ca8653cb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682416536817 2023-04-25 09:55:39,758 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 979a9c4af53a4b5995e0412455e65861, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682416538039 2023-04-25 09:55:39,795 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#B#compaction#382 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:39,815 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:39,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:39,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:39,844 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#A#compaction#381 average throughput is 0.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:39,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:39,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:39,867 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:39,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:39,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:39,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:39,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:39,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:39,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:39,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:39,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:39,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:39,892 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:39,892 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:39,892 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:39,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:39,907 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:39,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:39,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:39,920 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-25 09:55:39,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:39,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:39,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:39,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:39,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:39,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:40,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:40,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:40,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416600145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416600150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416600150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416600151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416600164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416600265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416600269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416600269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416600270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416600282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,294 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/25e4a9eb898e40fab09e869f4fc11dc9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/25e4a9eb898e40fab09e869f4fc11dc9 2023-04-25 09:55:40,298 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/B of 2ce6230fec57b1965ac37951c8640566 into 25e4a9eb898e40fab09e869f4fc11dc9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:40,298 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:40,298 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/B, priority=9, startTime=1682416539744; duration=0sec 2023-04-25 09:55:40,299 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:40,299 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-25 09:55:40,301 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84928 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-25 09:55:40,301 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/C is initiating minor compaction (all files) 2023-04-25 09:55:40,301 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/C in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:40,301 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7f1f4623740447a59f397454708f9a7f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4433d0ae66804146b58b6e80efaa65ac, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/6e4dec0570984e5ba01835dea751a4b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/abf8aac53ede4c9fb363c79424a46a47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d3c63184a97842c7872117fc0c591edf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c7921b0c945349d4a878ffe66c366f96, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d5682afcf1d94470bd6c8e3d4b6a964f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=82.9 K 2023-04-25 09:55:40,301 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7f1f4623740447a59f397454708f9a7f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1682416531459 2023-04-25 09:55:40,302 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4433d0ae66804146b58b6e80efaa65ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682416531937 2023-04-25 09:55:40,302 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6e4dec0570984e5ba01835dea751a4b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1682416533653 2023-04-25 09:55:40,302 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting abf8aac53ede4c9fb363c79424a46a47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1682416534406 2023-04-25 09:55:40,303 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d3c63184a97842c7872117fc0c591edf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1682416535585 2023-04-25 09:55:40,303 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c7921b0c945349d4a878ffe66c366f96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1682416536817 2023-04-25 09:55:40,303 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d5682afcf1d94470bd6c8e3d4b6a964f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682416538039 2023-04-25 09:55:40,317 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#C#compaction#384 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:40,339 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4cbd86b26d00469486ddc839092a3a81 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4cbd86b26d00469486ddc839092a3a81 2023-04-25 09:55:40,346 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/C of 2ce6230fec57b1965ac37951c8640566 into 4cbd86b26d00469486ddc839092a3a81(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:40,346 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:40,346 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/C, priority=9, startTime=1682416539745; duration=0sec 2023-04-25 09:55:40,346 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:40,388 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425a187d146242f4fba8011774d74af8648 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a187d146242f4fba8011774d74af8648 2023-04-25 09:55:40,443 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/3f0a5d45b936457f91fcc1af5fa213b1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/3f0a5d45b936457f91fcc1af5fa213b1 2023-04-25 09:55:40,446 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/A of 2ce6230fec57b1965ac37951c8640566 into 3f0a5d45b936457f91fcc1af5fa213b1(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:40,446 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:40,447 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/A, priority=9, startTime=1682416539744; duration=0sec 2023-04-25 09:55:40,447 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:40,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416600475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416600482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416600482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416600483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416600492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416600783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416600790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416600790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416600791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:40,799 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=296, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/5a2200b335bc40b7a241e1149b863f9a 2023-04-25 09:55:40,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:40,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416600800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,211 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/280b0035c952462eba1fa675b21ea111 2023-04-25 09:55:41,244 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4d01a915a14e45cb88f476bb6e35c154 2023-04-25 09:55:41,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/5a2200b335bc40b7a241e1149b863f9a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5a2200b335bc40b7a241e1149b863f9a 2023-04-25 09:55:41,256 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5a2200b335bc40b7a241e1149b863f9a, entries=150, sequenceid=296, filesize=25.4 K 2023-04-25 09:55:41,257 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/280b0035c952462eba1fa675b21ea111 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/280b0035c952462eba1fa675b21ea111 2023-04-25 09:55:41,270 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/280b0035c952462eba1fa675b21ea111, entries=150, sequenceid=296, filesize=11.9 K 2023-04-25 09:55:41,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4d01a915a14e45cb88f476bb6e35c154 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4d01a915a14e45cb88f476bb6e35c154 2023-04-25 09:55:41,281 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4d01a915a14e45cb88f476bb6e35c154, entries=150, sequenceid=296, filesize=11.9 K 2023-04-25 09:55:41,282 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2ce6230fec57b1965ac37951c8640566 in 1362ms, sequenceid=296, compaction requested=false 2023-04-25 09:55:41,282 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:41,282 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:41,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:41,282 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:41,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:41,286 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,286 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a860ef[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:41,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:41,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:41,289 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:41,291 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:41,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:41,293 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:41,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:41,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:41,293 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:41,293 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:41,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:41,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:41,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,303 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,305 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:41,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:41,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:41,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,310 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:41,310 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416601324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416601324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416601325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416601326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416601335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,356 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f94875e3727a46dc861df7548d7bdd1e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f94875e3727a46dc861df7548d7bdd1e 2023-04-25 09:55:41,378 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/01b51f59b26143e28f0c926f84ce27a6 2023-04-25 09:55:41,403 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/7905e88965a94426b6ea85a2381538cd 2023-04-25 09:55:41,410 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:41,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:41,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:41,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:41,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:41,417 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:41,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:41,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:41,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:41,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:41,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:41,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:41,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:41,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:41,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:41,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:41,437 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,437 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:41,437 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:41,437 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:41,439 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d3efff[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:41,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:41,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416601436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416601436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416601436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:41,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:41,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:41,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416601438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:41,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:41,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:41,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:41,448 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:41,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:41,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:41,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:41,448 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:41,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416601446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:41,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:41,464 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,464 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,466 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:41,466 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:41,566 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:41,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:41,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:41,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:41,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:41,569 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:41,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:41,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:41,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,572 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:41,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:41,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:41,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:41,574 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:41,574 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,574 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:41,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,585 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,585 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:41,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6081c654[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:41,586 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:41,586 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,587 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:41,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:41,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:41,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:41,591 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:41,593 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:41,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:41,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:41,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:41,593 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:41,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,611 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,611 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:41,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,615 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,615 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:41,615 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:41,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416601643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416601643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416601643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416601647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416601653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,716 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:41,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:41,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:41,719 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:41,719 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,719 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:41,719 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:41,720 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,720 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,721 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,721 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:41,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:41,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:41,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:41,722 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:41,722 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,723 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:41,724 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,724 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:41,724 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:41,724 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:41,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:41,726 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:41,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:41,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c081ac0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:41,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:41,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,731 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:41,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:41,733 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:41,733 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,733 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:41,733 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:41,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:41,737 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:41,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:41,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:41,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:41,737 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:41,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,740 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:41,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,741 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:41,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:41,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:41,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:41,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:41,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:41,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,750 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:41,750 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:41,749 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:41,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:41,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:41,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/1b8c2dbd84674c31b3e053f899be0c19 2023-04-25 09:55:41,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/01b51f59b26143e28f0c926f84ce27a6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/01b51f59b26143e28f0c926f84ce27a6 2023-04-25 09:55:41,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/01b51f59b26143e28f0c926f84ce27a6, entries=200, sequenceid=316, filesize=32.3 K 2023-04-25 09:55:41,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/7905e88965a94426b6ea85a2381538cd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/7905e88965a94426b6ea85a2381538cd 2023-04-25 09:55:41,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/7905e88965a94426b6ea85a2381538cd, entries=150, sequenceid=316, filesize=11.9 K 2023-04-25 09:55:41,850 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:41,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/1b8c2dbd84674c31b3e053f899be0c19 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b8c2dbd84674c31b3e053f899be0c19 2023-04-25 09:55:41,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:41,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:41,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:41,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:41,854 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:41,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:41,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:41,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,856 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:41,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:41,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:41,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:41,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b8c2dbd84674c31b3e053f899be0c19, entries=150, sequenceid=316, filesize=11.9 K 2023-04-25 09:55:41,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ce6230fec57b1965ac37951c8640566 in 567ms, sequenceid=316, compaction requested=true 2023-04-25 09:55:41,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:41,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:41,863 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:41,863 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:41,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:41,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:41,863 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85418 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:41,863 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/A is initiating minor compaction (all files) 2023-04-25 09:55:41,864 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:41,864 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/A in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,864 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/B is initiating minor compaction (all files) 2023-04-25 09:55:41,864 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/3f0a5d45b936457f91fcc1af5fa213b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5a2200b335bc40b7a241e1149b863f9a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/01b51f59b26143e28f0c926f84ce27a6] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=83.4 K 2023-04-25 09:55:41,864 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/B in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:41,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:41,864 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/25e4a9eb898e40fab09e869f4fc11dc9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/280b0035c952462eba1fa675b21ea111, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/7905e88965a94426b6ea85a2381538cd] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=35.9 K 2023-04-25 09:55:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:41,864 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3f0a5d45b936457f91fcc1af5fa213b1, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682416538039 2023-04-25 09:55:41,864 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 25e4a9eb898e40fab09e869f4fc11dc9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682416538039 2023-04-25 09:55:41,865 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5a2200b335bc40b7a241e1149b863f9a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682416538840 2023-04-25 09:55:41,865 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 280b0035c952462eba1fa675b21ea111, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682416538840 2023-04-25 09:55:41,865 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 01b51f59b26143e28f0c926f84ce27a6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682416540142 2023-04-25 09:55:41,865 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7905e88965a94426b6ea85a2381538cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682416540147 2023-04-25 09:55:41,878 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#B#compaction#390 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:41,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:41,904 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:41,904 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:41,904 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:55:41,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:41,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:41,905 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:41,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:41,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:41,907 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:41,937 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#A#compaction#391 average throughput is 0.37 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:41,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:41,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:41,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416601973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416601976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416601977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:41,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416601979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:41,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416601983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,016 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425cd769f9feb8544c5a3797260f68be791 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cd769f9feb8544c5a3797260f68be791 2023-04-25 09:55:42,034 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=331, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/8d34425b5c8e47c1a93a02366b5f55e2 2023-04-25 09:55:42,070 WARN [DataStreamer for file /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/472cd49a45d848e8b702c1e4aecb83ec block BP-1066761539-172.31.14.131-1682416412467:blk_1073742306_1482] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.closeInternal(DataStreamer.java:844) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:840) 2023-04-25 09:55:42,070 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/472cd49a45d848e8b702c1e4aecb83ec 2023-04-25 09:55:42,088 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/132770ae41b14ccaa9f80199824208bc 2023-04-25 09:55:42,094 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/8d34425b5c8e47c1a93a02366b5f55e2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8d34425b5c8e47c1a93a02366b5f55e2 2023-04-25 09:55:42,097 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8d34425b5c8e47c1a93a02366b5f55e2, entries=150, sequenceid=331, filesize=25.4 K 2023-04-25 09:55:42,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416602085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416602085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416602091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,099 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/472cd49a45d848e8b702c1e4aecb83ec as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/472cd49a45d848e8b702c1e4aecb83ec 2023-04-25 09:55:42,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416602092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416602101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,115 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/472cd49a45d848e8b702c1e4aecb83ec, entries=150, sequenceid=331, filesize=11.9 K 2023-04-25 09:55:42,116 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/132770ae41b14ccaa9f80199824208bc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/132770ae41b14ccaa9f80199824208bc 2023-04-25 09:55:42,119 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/132770ae41b14ccaa9f80199824208bc, entries=150, sequenceid=331, filesize=11.9 K 2023-04-25 09:55:42,120 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ce6230fec57b1965ac37951c8640566 in 216ms, sequenceid=331, compaction requested=true 2023-04-25 09:55:42,120 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:42,120 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:42,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:42,122 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:42,122 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:42,123 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,123 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:42,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:42,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e65a0b6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:42,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:42,124 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,125 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:42,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:42,126 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:42,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:42,128 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:42,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:42,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:42,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:42,129 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:42,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:42,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:42,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:42,129 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:42,131 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,131 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:42,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,131 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:42,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:42,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,135 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:42,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:42,138 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:42,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:42,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:42,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:42,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:42,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:42,139 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:42,139 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:42,239 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:42,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:42,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:42,243 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:42,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:42,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:42,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:42,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:42,244 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:42,244 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:42,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:42,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:42,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:42,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:42,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:42,247 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:42,247 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:42,247 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:42,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:42,248 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:42,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:42,249 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:42,249 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:42,249 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:42,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:42,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:42,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416602319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416602323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416602320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416602323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416602324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,336 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/a8bec0a8b37a4f01a002a3fd6c3ceba4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/a8bec0a8b37a4f01a002a3fd6c3ceba4 2023-04-25 09:55:42,340 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/B of 2ce6230fec57b1965ac37951c8640566 into a8bec0a8b37a4f01a002a3fd6c3ceba4(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:42,340 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:42,340 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/B, priority=13, startTime=1682416541863; duration=0sec 2023-04-25 09:55:42,340 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:42,340 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:42,341 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:42,342 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/C is initiating minor compaction (all files) 2023-04-25 09:55:42,342 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/C in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:42,342 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4cbd86b26d00469486ddc839092a3a81, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4d01a915a14e45cb88f476bb6e35c154, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b8c2dbd84674c31b3e053f899be0c19, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/132770ae41b14ccaa9f80199824208bc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=47.8 K 2023-04-25 09:55:42,342 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4cbd86b26d00469486ddc839092a3a81, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682416538039 2023-04-25 09:55:42,343 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4d01a915a14e45cb88f476bb6e35c154, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1682416538840 2023-04-25 09:55:42,343 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1b8c2dbd84674c31b3e053f899be0c19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682416540147 2023-04-25 09:55:42,343 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 132770ae41b14ccaa9f80199824208bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416541323 2023-04-25 09:55:42,356 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#C#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:42,398 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/364c27d04da743deb99fa40da09f3921 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/364c27d04da743deb99fa40da09f3921 2023-04-25 09:55:42,402 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/A of 2ce6230fec57b1965ac37951c8640566 into 364c27d04da743deb99fa40da09f3921(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:42,402 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:42,402 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/A, priority=13, startTime=1682416541862; duration=0sec 2023-04-25 09:55:42,402 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:42,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416602426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416602432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416602435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416602435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416602435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416602631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416602643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416602643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416602643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416602644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,669 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425ebbedc9b0f8b43f48260788621fe5f3a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ebbedc9b0f8b43f48260788621fe5f3a 2023-04-25 09:55:42,774 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/e87dfd72381a43fbab1ce653b88d91d9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/e87dfd72381a43fbab1ce653b88d91d9 2023-04-25 09:55:42,778 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/C of 2ce6230fec57b1965ac37951c8640566 into e87dfd72381a43fbab1ce653b88d91d9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:42,778 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:42,778 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/C, priority=12, startTime=1682416541863; duration=0sec 2023-04-25 09:55:42,778 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:42,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416602936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416602944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416602945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416602946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:42,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:42,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416602947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,077 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/83b4d0f56ce34e20b727753673c9c9f0 2023-04-25 09:55:43,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416603446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:43,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416603452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:43,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416603452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:43,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416603452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:43,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416603455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,488 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/ff7e69605736442baea505a45489981f 2023-04-25 09:55:43,898 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/42044415ae264cfc83a25621389acd00 2023-04-25 09:55:43,902 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/83b4d0f56ce34e20b727753673c9c9f0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/83b4d0f56ce34e20b727753673c9c9f0 2023-04-25 09:55:43,906 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/83b4d0f56ce34e20b727753673c9c9f0, entries=150, sequenceid=348, filesize=25.4 K 2023-04-25 09:55:43,906 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/ff7e69605736442baea505a45489981f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/ff7e69605736442baea505a45489981f 2023-04-25 09:55:43,910 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/ff7e69605736442baea505a45489981f, entries=150, sequenceid=348, filesize=11.9 K 2023-04-25 09:55:43,910 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/42044415ae264cfc83a25621389acd00 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/42044415ae264cfc83a25621389acd00 2023-04-25 09:55:43,913 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/42044415ae264cfc83a25621389acd00, entries=150, sequenceid=348, filesize=11.9 K 2023-04-25 09:55:43,914 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ce6230fec57b1965ac37951c8640566 in 1666ms, sequenceid=348, compaction requested=true 2023-04-25 09:55:43,914 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:43,914 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:43,914 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:43,914 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:43,914 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,914 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:43,915 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:43,916 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,917 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,917 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:43,917 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294e0e92[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:43,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:43,919 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:43,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:43,920 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,920 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:43,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:43,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:43,922 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:43,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:43,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:43,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:43,925 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:43,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:43,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:43,925 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:43,925 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:43,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:43,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:43,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:43,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:43,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:43,933 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:43,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:43,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:43,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:43,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:43,933 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:43,935 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:43,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:43,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:43,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:43,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:43,935 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:43,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:44,035 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:44,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:44,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:44,039 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:44,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:44,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:44,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:44,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:44,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:44,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:44,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:44,042 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:44,043 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:44,043 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:44,059 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255822179fe8704835a1798763b573a57f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255822179fe8704835a1798763b573a57f 2023-04-25 09:55:44,065 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/73929cd6783249d78f1f4dce1739a652 2023-04-25 09:55:44,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:44,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:44,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416604474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416604475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416604475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416604476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,481 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/8157fcf626414fe9b63ecfa697f8b435 2023-04-25 09:55:44,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416604480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,490 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/2391d1c0b8274bafb7391c3013d18daf 2023-04-25 09:55:44,498 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/73929cd6783249d78f1f4dce1739a652 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/73929cd6783249d78f1f4dce1739a652 2023-04-25 09:55:44,501 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/73929cd6783249d78f1f4dce1739a652, entries=150, sequenceid=366, filesize=25.4 K 2023-04-25 09:55:44,502 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/8157fcf626414fe9b63ecfa697f8b435 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/8157fcf626414fe9b63ecfa697f8b435 2023-04-25 09:55:44,505 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/8157fcf626414fe9b63ecfa697f8b435, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 09:55:44,505 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/2391d1c0b8274bafb7391c3013d18daf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/2391d1c0b8274bafb7391c3013d18daf 2023-04-25 09:55:44,508 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/2391d1c0b8274bafb7391c3013d18daf, entries=150, sequenceid=366, filesize=11.9 K 2023-04-25 09:55:44,509 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ce6230fec57b1965ac37951c8640566 in 466ms, sequenceid=366, compaction requested=true 2023-04-25 09:55:44,509 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:44,509 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:44,509 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:44,509 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:44,509 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,511 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:44,511 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:44,512 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4713de89[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:44,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:44,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,518 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:44,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:44,519 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:44,519 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,519 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:44,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,520 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:44,520 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:44,520 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:44,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:44,523 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:44,523 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:44,523 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:44,523 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:44,523 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:44,524 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,525 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,525 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:44,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:44,526 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:44,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:44,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:44,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:44,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:44,540 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:44,540 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:44,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:44,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-25 09:55:44,593 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:44,593 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:44,593 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:44,593 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:44,593 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:44,593 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:44,615 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e417d9d8ed254f009dcb4c32b892de87 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e417d9d8ed254f009dcb4c32b892de87 2023-04-25 09:55:44,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416604606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416604610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416604613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416604618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416604619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,640 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:44,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:44,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:44,652 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:44,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:44,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:44,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:44,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,652 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,653 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:44,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:44,654 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,654 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:44,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:44,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:44,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:44,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:44,655 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:44,656 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:44,656 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. started... 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:44,656 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. as already flushing 2023-04-25 09:55:44,656 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:44,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,659 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:44,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:44,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6be102b1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:44,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:44,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:44,671 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:44,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:44,677 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:44,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:44,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:44,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:44,677 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:44,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:44,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:44,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:44,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:44,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:44,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:44,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:44,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:44,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:44,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:44,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:44,688 DEBUG [Thread-2491] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:44,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:44,688 DEBUG [Thread-2491] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:44,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:44,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:44,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416604721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43052 deadline: 1682416604722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416604722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416604732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416604733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,757 DEBUG [Thread-2498] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:60014 2023-04-25 09:55:44,757 DEBUG [Thread-2498] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:44,760 DEBUG [Thread-2494] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:60014 2023-04-25 09:55:44,760 DEBUG [Thread-2494] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:44,765 DEBUG [Thread-2496] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:60014 2023-04-25 09:55:44,766 DEBUG [Thread-2496] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:44,767 DEBUG [Thread-2492] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:60014 2023-04-25 09:55:44,767 DEBUG [Thread-2492] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:44,771 DEBUG [Thread-2500] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:60014 2023-04-25 09:55:44,771 DEBUG [Thread-2500] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:44,788 DEBUG [Thread-2491] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:44,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:44,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43052 deadline: 1682416604932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416604932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416604933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416604941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:44,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:44,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416604942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,035 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=386, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/8b4c8b130c5943a38ad33c9a014db0b3 2023-04-25 09:55:45,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43052 deadline: 1682416605235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416605235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416605235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416605244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416605245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/60b0ecba4a7e4798a69ee19b8a55fe7b 2023-04-25 09:55:45,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43052 deadline: 1682416605736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43072 deadline: 1682416605739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:43010 deadline: 1682416605740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43026 deadline: 1682416605746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:45,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:43060 deadline: 1682416605749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:45,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/30f16005ec154dbea03c805135738d1b 2023-04-25 09:55:45,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/8b4c8b130c5943a38ad33c9a014db0b3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8b4c8b130c5943a38ad33c9a014db0b3 2023-04-25 09:55:45,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8b4c8b130c5943a38ad33c9a014db0b3, entries=150, sequenceid=386, filesize=25.4 K 2023-04-25 09:55:45,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/60b0ecba4a7e4798a69ee19b8a55fe7b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/60b0ecba4a7e4798a69ee19b8a55fe7b 2023-04-25 09:55:45,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/60b0ecba4a7e4798a69ee19b8a55fe7b, entries=150, sequenceid=386, filesize=11.9 K 2023-04-25 09:55:45,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/30f16005ec154dbea03c805135738d1b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/30f16005ec154dbea03c805135738d1b 2023-04-25 09:55:45,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/30f16005ec154dbea03c805135738d1b, entries=150, sequenceid=386, filesize=11.9 K 2023-04-25 09:55:45,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2ce6230fec57b1965ac37951c8640566 in 1274ms, sequenceid=386, compaction requested=true 2023-04-25 09:55:45,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:45,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:45,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:45,866 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:55:45,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:45,866 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:55:45,867 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130572 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:55:45,867 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:55:45,867 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/A is initiating minor compaction (all files) 2023-04-25 09:55:45,867 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/B is initiating minor compaction (all files) 2023-04-25 09:55:45,868 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/A in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:45,868 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/B in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:45,868 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/364c27d04da743deb99fa40da09f3921, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8d34425b5c8e47c1a93a02366b5f55e2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/83b4d0f56ce34e20b727753673c9c9f0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/73929cd6783249d78f1f4dce1739a652, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8b4c8b130c5943a38ad33c9a014db0b3] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=127.5 K 2023-04-25 09:55:45,868 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/a8bec0a8b37a4f01a002a3fd6c3ceba4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/472cd49a45d848e8b702c1e4aecb83ec, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/ff7e69605736442baea505a45489981f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/8157fcf626414fe9b63ecfa697f8b435, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/60b0ecba4a7e4798a69ee19b8a55fe7b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=59.8 K 2023-04-25 09:55:45,868 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 364c27d04da743deb99fa40da09f3921, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682416540147 2023-04-25 09:55:45,868 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a8bec0a8b37a4f01a002a3fd6c3ceba4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1682416540147 2023-04-25 09:55:45,868 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8d34425b5c8e47c1a93a02366b5f55e2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416541323 2023-04-25 09:55:45,868 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 472cd49a45d848e8b702c1e4aecb83ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416541323 2023-04-25 09:55:45,868 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 83b4d0f56ce34e20b727753673c9c9f0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682416541963 2023-04-25 09:55:45,868 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ff7e69605736442baea505a45489981f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682416541963 2023-04-25 09:55:45,869 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 73929cd6783249d78f1f4dce1739a652, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682416542318 2023-04-25 09:55:45,869 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8157fcf626414fe9b63ecfa697f8b435, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682416542318 2023-04-25 09:55:45,869 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8b4c8b130c5943a38ad33c9a014db0b3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682416544479 2023-04-25 09:55:45,869 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 60b0ecba4a7e4798a69ee19b8a55fe7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682416544479 2023-04-25 09:55:45,876 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:45,878 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#A#compaction#405 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:46,284 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/29f60aef2efc46af8d84bec11140b841 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/29f60aef2efc46af8d84bec11140b841 2023-04-25 09:55:46,288 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/B of 2ce6230fec57b1965ac37951c8640566 into 29f60aef2efc46af8d84bec11140b841(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:46,288 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:46,288 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/B, priority=11, startTime=1682416545866; duration=0sec 2023-04-25 09:55:46,288 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:46,289 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:46,289 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:46,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 2ce6230fec57b1965ac37951c8640566/C is initiating minor compaction (all files) 2023-04-25 09:55:46,290 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ce6230fec57b1965ac37951c8640566/C in TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:46,290 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/e87dfd72381a43fbab1ce653b88d91d9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/42044415ae264cfc83a25621389acd00, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/2391d1c0b8274bafb7391c3013d18daf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/30f16005ec154dbea03c805135738d1b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp, totalSize=48.0 K 2023-04-25 09:55:46,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e87dfd72381a43fbab1ce653b88d91d9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1682416541323 2023-04-25 09:55:46,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 42044415ae264cfc83a25621389acd00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1682416541963 2023-04-25 09:55:46,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2391d1c0b8274bafb7391c3013d18daf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682416542318 2023-04-25 09:55:46,290 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 30f16005ec154dbea03c805135738d1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682416544479 2023-04-25 09:55:46,296 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ce6230fec57b1965ac37951c8640566#C#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:46,703 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/d6ecd1425496470286191405b6358516 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d6ecd1425496470286191405b6358516 2023-04-25 09:55:46,707 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/C of 2ce6230fec57b1965ac37951c8640566 into d6ecd1425496470286191405b6358516(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:46,707 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:46,707 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/C, priority=12, startTime=1682416545866; duration=0sec 2023-04-25 09:55:46,707 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:46,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 09:55:46,743 DEBUG [Thread-2487] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:60014 2023-04-25 09:55:46,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:46,743 DEBUG [Thread-2487] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:46,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:46,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:46,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:46,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:46,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:46,745 DEBUG [Thread-2489] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:60014 2023-04-25 09:55:46,745 DEBUG [Thread-2489] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:46,751 DEBUG [Thread-2485] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:60014 2023-04-25 09:55:46,751 DEBUG [Thread-2485] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:46,766 DEBUG [Thread-2481] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:60014 2023-04-25 09:55:46,766 DEBUG [Thread-2483] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:60014 2023-04-25 09:55:46,766 DEBUG [Thread-2481] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:46,766 DEBUG [Thread-2483] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:46,766 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1642 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 4926 rows 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1652 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 4952 rows 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1640 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 4919 rows 2023-04-25 09:55:46,767 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1634 2023-04-25 09:55:46,768 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 4902 rows 2023-04-25 09:55:46,768 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1643 2023-04-25 09:55:46,768 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 4928 rows 2023-04-25 09:55:46,768 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:55:46,768 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:60014 2023-04-25 09:55:46,768 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:55:46,769 INFO [Listener at localhost/33705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 09:55:46,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 09:55:46,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:46,774 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416546774"}]},"ts":"1682416546774"} 2023-04-25 09:55:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:46,775 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 09:55:46,777 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 09:55:46,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, UNASSIGN}] 2023-04-25 09:55:46,779 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, UNASSIGN 2023-04-25 09:55:46,779 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:46,780 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416546779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416546779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416546779"}]},"ts":"1682416546779"} 2023-04-25 09:55:46,781 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:55:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:46,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:46,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2ce6230fec57b1965ac37951c8640566, disabling compactions & flushes 2023-04-25 09:55:46,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:47,098 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/a122f434cd3c42a0b27157b7dc88ebb6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a122f434cd3c42a0b27157b7dc88ebb6 2023-04-25 09:55:47,101 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ce6230fec57b1965ac37951c8640566/A of 2ce6230fec57b1965ac37951c8640566 into a122f434cd3c42a0b27157b7dc88ebb6(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 09:55:47,101 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:47,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:47,101 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566., storeName=2ce6230fec57b1965ac37951c8640566/A, priority=11, startTime=1682416545866; duration=1sec 2023-04-25 09:55:47,101 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:47,160 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042598b2f45141fe4bb3bf7bc45ae1a11fe2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042598b2f45141fe4bb3bf7bc45ae1a11fe2 2023-04-25 09:55:47,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:47,564 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/94f795f0452d44bca440e5265cbcf62e 2023-04-25 09:55:47,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/191fa19642f640a3925609e57bf4fc53 2023-04-25 09:55:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:47,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/ad3efe2b81db4c2a96c83272c89df266 2023-04-25 09:55:47,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/94f795f0452d44bca440e5265cbcf62e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/94f795f0452d44bca440e5265cbcf62e 2023-04-25 09:55:47,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/94f795f0452d44bca440e5265cbcf62e, entries=150, sequenceid=401, filesize=25.4 K 2023-04-25 09:55:47,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/191fa19642f640a3925609e57bf4fc53 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/191fa19642f640a3925609e57bf4fc53 2023-04-25 09:55:47,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/191fa19642f640a3925609e57bf4fc53, entries=150, sequenceid=401, filesize=11.9 K 2023-04-25 09:55:47,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/ad3efe2b81db4c2a96c83272c89df266 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ad3efe2b81db4c2a96c83272c89df266 2023-04-25 09:55:47,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ad3efe2b81db4c2a96c83272c89df266, entries=150, sequenceid=401, filesize=11.9 K 2023-04-25 09:55:47,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=26.84 KB/27480 for 2ce6230fec57b1965ac37951c8640566 in 1255ms, sequenceid=401, compaction requested=false 2023-04-25 09:55:47,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:47,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. after waiting 0 ms 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:47,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2ce6230fec57b1965ac37951c8640566 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=A 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=B 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ce6230fec57b1965ac37951c8640566, store=C 2023-04-25 09:55:47,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:48,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257ed1d6782bbb40d2a56e905455580a6c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257ed1d6782bbb40d2a56e905455580a6c 2023-04-25 09:55:48,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/f227bb4f589e4b4a93d3160229e4261a 2023-04-25 09:55:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:49,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/1c035d84017d4a1987baf2e77efa7021 2023-04-25 09:55:49,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4e1b6c6e54a44d38862750a531800310 2023-04-25 09:55:49,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/A/f227bb4f589e4b4a93d3160229e4261a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/f227bb4f589e4b4a93d3160229e4261a 2023-04-25 09:55:49,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/f227bb4f589e4b4a93d3160229e4261a, entries=150, sequenceid=409, filesize=25.4 K 2023-04-25 09:55:49,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/B/1c035d84017d4a1987baf2e77efa7021 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1c035d84017d4a1987baf2e77efa7021 2023-04-25 09:55:49,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1c035d84017d4a1987baf2e77efa7021, entries=150, sequenceid=409, filesize=11.9 K 2023-04-25 09:55:49,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/.tmp/C/4e1b6c6e54a44d38862750a531800310 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4e1b6c6e54a44d38862750a531800310 2023-04-25 09:55:49,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4e1b6c6e54a44d38862750a531800310, entries=150, sequenceid=409, filesize=11.9 K 2023-04-25 09:55:49,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2ce6230fec57b1965ac37951c8640566 in 1645ms, sequenceid=409, compaction requested=true 2023-04-25 09:55:49,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a8f1a31d17f042378f6d3c5bb606ee01, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b25cad25efaa44a78a1e2c547b1ea641, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/1d59b4fd126c4d98afef637ade930a8b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/696ebc63d2ef4b63b26d69aa6afb8af0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/4aa2bd12623a477eb59b236bfe65b82a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/ec23cfce2442407285a3e1f1ca8653cb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/979a9c4af53a4b5995e0412455e65861, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/3f0a5d45b936457f91fcc1af5fa213b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5a2200b335bc40b7a241e1149b863f9a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/01b51f59b26143e28f0c926f84ce27a6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/364c27d04da743deb99fa40da09f3921, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8d34425b5c8e47c1a93a02366b5f55e2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/83b4d0f56ce34e20b727753673c9c9f0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/73929cd6783249d78f1f4dce1739a652, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8b4c8b130c5943a38ad33c9a014db0b3] to archive 2023-04-25 09:55:49,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:49,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a8f1a31d17f042378f6d3c5bb606ee01 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a8f1a31d17f042378f6d3c5bb606ee01 2023-04-25 09:55:49,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b25cad25efaa44a78a1e2c547b1ea641 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/b25cad25efaa44a78a1e2c547b1ea641 2023-04-25 09:55:49,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/1d59b4fd126c4d98afef637ade930a8b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/1d59b4fd126c4d98afef637ade930a8b 2023-04-25 09:55:49,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/696ebc63d2ef4b63b26d69aa6afb8af0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/696ebc63d2ef4b63b26d69aa6afb8af0 2023-04-25 09:55:49,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/4aa2bd12623a477eb59b236bfe65b82a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/4aa2bd12623a477eb59b236bfe65b82a 2023-04-25 09:55:49,648 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/ec23cfce2442407285a3e1f1ca8653cb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/ec23cfce2442407285a3e1f1ca8653cb 2023-04-25 09:55:49,649 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/979a9c4af53a4b5995e0412455e65861 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/979a9c4af53a4b5995e0412455e65861 2023-04-25 09:55:49,650 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/3f0a5d45b936457f91fcc1af5fa213b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/3f0a5d45b936457f91fcc1af5fa213b1 2023-04-25 09:55:49,650 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5a2200b335bc40b7a241e1149b863f9a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/5a2200b335bc40b7a241e1149b863f9a 2023-04-25 09:55:49,651 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/01b51f59b26143e28f0c926f84ce27a6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/01b51f59b26143e28f0c926f84ce27a6 2023-04-25 09:55:49,652 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/364c27d04da743deb99fa40da09f3921 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/364c27d04da743deb99fa40da09f3921 2023-04-25 09:55:49,653 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8d34425b5c8e47c1a93a02366b5f55e2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8d34425b5c8e47c1a93a02366b5f55e2 2023-04-25 09:55:49,653 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/83b4d0f56ce34e20b727753673c9c9f0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/83b4d0f56ce34e20b727753673c9c9f0 2023-04-25 09:55:49,654 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/73929cd6783249d78f1f4dce1739a652 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/73929cd6783249d78f1f4dce1739a652 2023-04-25 09:55:49,655 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8b4c8b130c5943a38ad33c9a014db0b3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/8b4c8b130c5943a38ad33c9a014db0b3 2023-04-25 09:55:49,657 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3f63ef9d10c74f09a8653d031a6a0257, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3d9acbd366bc43db87fd5b12d1f682fc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1a03571ede2a4ccaa0a908b1b6b9ea83, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/933e7bd0de9c4dcfba23d4620ce721a3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/2e263355313c457889605d5b11be1d75, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/328aa63be66e4aa5a8dd90eef807bbcf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/25e4a9eb898e40fab09e869f4fc11dc9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/bbb3c5f0892843f2a3772771496561d7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/280b0035c952462eba1fa675b21ea111, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/a8bec0a8b37a4f01a002a3fd6c3ceba4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/7905e88965a94426b6ea85a2381538cd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/472cd49a45d848e8b702c1e4aecb83ec, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/ff7e69605736442baea505a45489981f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/8157fcf626414fe9b63ecfa697f8b435, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/60b0ecba4a7e4798a69ee19b8a55fe7b] to archive 2023-04-25 09:55:49,657 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:49,658 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3f63ef9d10c74f09a8653d031a6a0257 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3f63ef9d10c74f09a8653d031a6a0257 2023-04-25 09:55:49,659 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3d9acbd366bc43db87fd5b12d1f682fc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/3d9acbd366bc43db87fd5b12d1f682fc 2023-04-25 09:55:49,660 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1a03571ede2a4ccaa0a908b1b6b9ea83 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1a03571ede2a4ccaa0a908b1b6b9ea83 2023-04-25 09:55:49,661 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/933e7bd0de9c4dcfba23d4620ce721a3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/933e7bd0de9c4dcfba23d4620ce721a3 2023-04-25 09:55:49,661 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/2e263355313c457889605d5b11be1d75 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/2e263355313c457889605d5b11be1d75 2023-04-25 09:55:49,662 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/328aa63be66e4aa5a8dd90eef807bbcf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/328aa63be66e4aa5a8dd90eef807bbcf 2023-04-25 09:55:49,663 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/25e4a9eb898e40fab09e869f4fc11dc9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/25e4a9eb898e40fab09e869f4fc11dc9 2023-04-25 09:55:49,664 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/bbb3c5f0892843f2a3772771496561d7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/bbb3c5f0892843f2a3772771496561d7 2023-04-25 09:55:49,665 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/280b0035c952462eba1fa675b21ea111 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/280b0035c952462eba1fa675b21ea111 2023-04-25 09:55:49,666 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/a8bec0a8b37a4f01a002a3fd6c3ceba4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/a8bec0a8b37a4f01a002a3fd6c3ceba4 2023-04-25 09:55:49,667 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/7905e88965a94426b6ea85a2381538cd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/7905e88965a94426b6ea85a2381538cd 2023-04-25 09:55:49,668 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/472cd49a45d848e8b702c1e4aecb83ec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/472cd49a45d848e8b702c1e4aecb83ec 2023-04-25 09:55:49,668 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/ff7e69605736442baea505a45489981f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/ff7e69605736442baea505a45489981f 2023-04-25 09:55:49,669 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/8157fcf626414fe9b63ecfa697f8b435 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/8157fcf626414fe9b63ecfa697f8b435 2023-04-25 09:55:49,670 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/60b0ecba4a7e4798a69ee19b8a55fe7b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/60b0ecba4a7e4798a69ee19b8a55fe7b 2023-04-25 09:55:49,672 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7f1f4623740447a59f397454708f9a7f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4433d0ae66804146b58b6e80efaa65ac, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/6e4dec0570984e5ba01835dea751a4b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/abf8aac53ede4c9fb363c79424a46a47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d3c63184a97842c7872117fc0c591edf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c7921b0c945349d4a878ffe66c366f96, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4cbd86b26d00469486ddc839092a3a81, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d5682afcf1d94470bd6c8e3d4b6a964f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4d01a915a14e45cb88f476bb6e35c154, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b8c2dbd84674c31b3e053f899be0c19, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/e87dfd72381a43fbab1ce653b88d91d9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/132770ae41b14ccaa9f80199824208bc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/42044415ae264cfc83a25621389acd00, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/2391d1c0b8274bafb7391c3013d18daf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/30f16005ec154dbea03c805135738d1b] to archive 2023-04-25 09:55:49,672 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:55:49,674 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7f1f4623740447a59f397454708f9a7f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/7f1f4623740447a59f397454708f9a7f 2023-04-25 09:55:49,674 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4433d0ae66804146b58b6e80efaa65ac to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4433d0ae66804146b58b6e80efaa65ac 2023-04-25 09:55:49,675 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/6e4dec0570984e5ba01835dea751a4b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/6e4dec0570984e5ba01835dea751a4b1 2023-04-25 09:55:49,676 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/abf8aac53ede4c9fb363c79424a46a47 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/abf8aac53ede4c9fb363c79424a46a47 2023-04-25 09:55:49,677 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d3c63184a97842c7872117fc0c591edf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d3c63184a97842c7872117fc0c591edf 2023-04-25 09:55:49,678 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c7921b0c945349d4a878ffe66c366f96 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/c7921b0c945349d4a878ffe66c366f96 2023-04-25 09:55:49,678 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4cbd86b26d00469486ddc839092a3a81 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4cbd86b26d00469486ddc839092a3a81 2023-04-25 09:55:49,679 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d5682afcf1d94470bd6c8e3d4b6a964f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d5682afcf1d94470bd6c8e3d4b6a964f 2023-04-25 09:55:49,680 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4d01a915a14e45cb88f476bb6e35c154 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4d01a915a14e45cb88f476bb6e35c154 2023-04-25 09:55:49,681 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b8c2dbd84674c31b3e053f899be0c19 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/1b8c2dbd84674c31b3e053f899be0c19 2023-04-25 09:55:49,682 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/e87dfd72381a43fbab1ce653b88d91d9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/e87dfd72381a43fbab1ce653b88d91d9 2023-04-25 09:55:49,682 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/132770ae41b14ccaa9f80199824208bc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/132770ae41b14ccaa9f80199824208bc 2023-04-25 09:55:49,683 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/42044415ae264cfc83a25621389acd00 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/42044415ae264cfc83a25621389acd00 2023-04-25 09:55:49,684 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/2391d1c0b8274bafb7391c3013d18daf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/2391d1c0b8274bafb7391c3013d18daf 2023-04-25 09:55:49,685 DEBUG [StoreCloser-TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/30f16005ec154dbea03c805135738d1b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/30f16005ec154dbea03c805135738d1b 2023-04-25 09:55:49,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/recovered.edits/412.seqid, newMaxSeqId=412, maxSeqId=4 2023-04-25 09:55:49,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566. 2023-04-25 09:55:49,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2ce6230fec57b1965ac37951c8640566: 2023-04-25 09:55:49,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:49,692 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=2ce6230fec57b1965ac37951c8640566, regionState=CLOSED 2023-04-25 09:55:49,692 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416549692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416549692"}]},"ts":"1682416549692"} 2023-04-25 09:55:49,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-25 09:55:49,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 2ce6230fec57b1965ac37951c8640566, server=jenkins-hbase4.apache.org,39407,1682416417123 in 2.9130 sec 2023-04-25 09:55:49,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-25 09:55:49,695 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ce6230fec57b1965ac37951c8640566, UNASSIGN in 2.9170 sec 2023-04-25 09:55:49,696 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416549696"}]},"ts":"1682416549696"} 2023-04-25 09:55:49,696 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 09:55:49,698 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 09:55:49,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9290 sec 2023-04-25 09:55:50,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-25 09:55:50,881 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-25 09:55:50,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 09:55:50,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:50,884 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-25 09:55:50,884 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:50,889 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:50,891 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/recovered.edits] 2023-04-25 09:55:50,893 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/94f795f0452d44bca440e5265cbcf62e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/94f795f0452d44bca440e5265cbcf62e 2023-04-25 09:55:50,894 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a122f434cd3c42a0b27157b7dc88ebb6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/a122f434cd3c42a0b27157b7dc88ebb6 2023-04-25 09:55:50,895 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/f227bb4f589e4b4a93d3160229e4261a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/A/f227bb4f589e4b4a93d3160229e4261a 2023-04-25 09:55:50,897 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/191fa19642f640a3925609e57bf4fc53 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/191fa19642f640a3925609e57bf4fc53 2023-04-25 09:55:50,898 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1c035d84017d4a1987baf2e77efa7021 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/1c035d84017d4a1987baf2e77efa7021 2023-04-25 09:55:50,899 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/29f60aef2efc46af8d84bec11140b841 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/B/29f60aef2efc46af8d84bec11140b841 2023-04-25 09:55:50,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4e1b6c6e54a44d38862750a531800310 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/4e1b6c6e54a44d38862750a531800310 2023-04-25 09:55:50,902 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ad3efe2b81db4c2a96c83272c89df266 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/ad3efe2b81db4c2a96c83272c89df266 2023-04-25 09:55:50,903 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d6ecd1425496470286191405b6358516 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/C/d6ecd1425496470286191405b6358516 2023-04-25 09:55:50,905 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/recovered.edits/412.seqid to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566/recovered.edits/412.seqid 2023-04-25 09:55:50,905 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/2ce6230fec57b1965ac37951c8640566 2023-04-25 09:55:50,905 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:55:50,906 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 09:55:50,906 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 09:55:50,908 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251348f6cd4a25493abf5f2ebd88c80d90 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251348f6cd4a25493abf5f2ebd88c80d90 2023-04-25 09:55:50,909 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251491d4c1732845a8bf878bf2c3c067b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251491d4c1732845a8bf878bf2c3c067b6 2023-04-25 09:55:50,910 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042521b4de78ac5048cb94c8b9ff3fb83749 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042521b4de78ac5048cb94c8b9ff3fb83749 2023-04-25 09:55:50,911 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253820635f595349d48a9e67965fefe6c6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253820635f595349d48a9e67965fefe6c6 2023-04-25 09:55:50,912 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254031b04df97e427ea1268e3564fd1e4b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254031b04df97e427ea1268e3564fd1e4b 2023-04-25 09:55:50,913 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425410ac0238e9b4b888ff2191d8e104f9e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425410ac0238e9b4b888ff2191d8e104f9e 2023-04-25 09:55:50,914 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254b23869345a544aca51f50c72ce2761c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304254b23869345a544aca51f50c72ce2761c 2023-04-25 09:55:50,915 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255822179fe8704835a1798763b573a57f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255822179fe8704835a1798763b573a57f 2023-04-25 09:55:50,916 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042569315a88d81f435d9d5082032a92cfbc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042569315a88d81f435d9d5082032a92cfbc 2023-04-25 09:55:50,917 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257ed1d6782bbb40d2a56e905455580a6c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257ed1d6782bbb40d2a56e905455580a6c 2023-04-25 09:55:50,918 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042598b2f45141fe4bb3bf7bc45ae1a11fe2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042598b2f45141fe4bb3bf7bc45ae1a11fe2 2023-04-25 09:55:50,919 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a187d146242f4fba8011774d74af8648 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a187d146242f4fba8011774d74af8648 2023-04-25 09:55:50,920 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a444683f945f4e13af9f181f5f0c52ec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425a444683f945f4e13af9f181f5f0c52ec 2023-04-25 09:55:50,921 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bf5e4409eae340c8bba72fdc19f07816 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bf5e4409eae340c8bba72fdc19f07816 2023-04-25 09:55:50,921 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c06329d087094f458d3f4b5a105414ec to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c06329d087094f458d3f4b5a105414ec 2023-04-25 09:55:50,922 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ca12857221a84a06b01738ba25bed068 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ca12857221a84a06b01738ba25bed068 2023-04-25 09:55:50,923 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cd769f9feb8544c5a3797260f68be791 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cd769f9feb8544c5a3797260f68be791 2023-04-25 09:55:50,924 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e417d9d8ed254f009dcb4c32b892de87 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e417d9d8ed254f009dcb4c32b892de87 2023-04-25 09:55:50,925 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e48056c9607541cc94e7b1e68e0ba77c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e48056c9607541cc94e7b1e68e0ba77c 2023-04-25 09:55:50,926 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ebbedc9b0f8b43f48260788621fe5f3a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ebbedc9b0f8b43f48260788621fe5f3a 2023-04-25 09:55:50,927 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ec85e42fa7bc420c825a4af9d2993517 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425ec85e42fa7bc420c825a4af9d2993517 2023-04-25 09:55:50,928 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f8e4cf071f674eaa92ab45b44d70db39 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f8e4cf071f674eaa92ab45b44d70db39 2023-04-25 09:55:50,929 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f94875e3727a46dc861df7548d7bdd1e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f94875e3727a46dc861df7548d7bdd1e 2023-04-25 09:55:50,929 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 09:55:50,931 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:50,932 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 09:55:50,934 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 09:55:50,935 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:50,935 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 09:55:50,935 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682416550935"}]},"ts":"9223372036854775807"} 2023-04-25 09:55:50,936 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 09:55:50,936 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2ce6230fec57b1965ac37951c8640566, NAME => 'TestAcidGuarantees,,1682416521669.2ce6230fec57b1965ac37951c8640566.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 09:55:50,936 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 09:55:50,936 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682416550936"}]},"ts":"9223372036854775807"} 2023-04-25 09:55:50,937 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 09:55:50,939 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:50,939 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-25 09:55:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-25 09:55:50,985 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-25 09:55:50,996 INFO [Listener at localhost/33705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 224) - Thread LEAK? -, OpenFileDescriptor=533 (was 530) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=372 (was 318) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=4410 (was 4054) - AvailableMemoryMB LEAK? - 2023-04-25 09:55:51,008 INFO [Listener at localhost/33705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=533, MaxFileDescriptor=60000, SystemLoadAverage=372, ProcessCount=170, AvailableMemoryMB=4409 2023-04-25 09:55:51,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:55:51,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:55:51,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 09:55:51,012 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:55:51,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-25 09:55:51,012 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:55:51,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 09:55:51,013 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:51,014 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 empty. 2023-04-25 09:55:51,014 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:51,014 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:55:51,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 09:55:51,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 09:55:51,424 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:55:51,425 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6b4247a1e1c946a4543dfbc0edb52631, NAME => 'TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:55:51,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 09:55:51,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:55:51,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6b4247a1e1c946a4543dfbc0edb52631, disabling compactions & flushes 2023-04-25 09:55:51,838 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:51,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:51,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. after waiting 0 ms 2023-04-25 09:55:51,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:51,839 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:51,839 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:51,840 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:55:51,841 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416551841"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416551841"}]},"ts":"1682416551841"} 2023-04-25 09:55:51,842 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:55:51,843 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:55:51,843 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416551843"}]},"ts":"1682416551843"} 2023-04-25 09:55:51,844 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 09:55:51,846 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, ASSIGN}] 2023-04-25 09:55:51,847 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, ASSIGN 2023-04-25 09:55:51,848 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:55:51,999 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=6b4247a1e1c946a4543dfbc0edb52631, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:51,999 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416551999"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416551999"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416551999"}]},"ts":"1682416551999"} 2023-04-25 09:55:52,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:55:52,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 09:55:52,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:52,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6b4247a1e1c946a4543dfbc0edb52631, NAME => 'TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:55:52,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:55:52,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,156 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,157 DEBUG [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A 2023-04-25 09:55:52,158 DEBUG [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A 2023-04-25 09:55:52,158 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:52,158 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b4247a1e1c946a4543dfbc0edb52631 columnFamilyName A 2023-04-25 09:55:52,158 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.HStore(310): Store=6b4247a1e1c946a4543dfbc0edb52631/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:52,158 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,159 DEBUG [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B 2023-04-25 09:55:52,159 DEBUG [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B 2023-04-25 09:55:52,160 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:52,160 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b4247a1e1c946a4543dfbc0edb52631 columnFamilyName B 2023-04-25 09:55:52,160 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.HStore(310): Store=6b4247a1e1c946a4543dfbc0edb52631/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:52,160 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,161 DEBUG [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C 2023-04-25 09:55:52,161 DEBUG [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C 2023-04-25 09:55:52,161 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:55:52,162 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b4247a1e1c946a4543dfbc0edb52631 columnFamilyName C 2023-04-25 09:55:52,162 INFO [StoreOpener-6b4247a1e1c946a4543dfbc0edb52631-1] regionserver.HStore(310): Store=6b4247a1e1c946a4543dfbc0edb52631/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:55:52,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:52,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:55:52,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:52,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:55:52,167 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6b4247a1e1c946a4543dfbc0edb52631; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70358604, jitterRate=0.04842489957809448}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:55:52,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:52,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., pid=49, masterSystemTime=1682416552152 2023-04-25 09:55:52,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:52,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:52,169 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=6b4247a1e1c946a4543dfbc0edb52631, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:52,169 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416552169"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416552169"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416552169"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416552169"}]},"ts":"1682416552169"} 2023-04-25 09:55:52,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-25 09:55:52,172 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 in 170 msec 2023-04-25 09:55:52,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-25 09:55:52,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, ASSIGN in 326 msec 2023-04-25 09:55:52,173 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:55:52,173 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416552173"}]},"ts":"1682416552173"} 2023-04-25 09:55:52,174 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 09:55:52,177 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:55:52,178 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1670 sec 2023-04-25 09:55:53,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-25 09:55:53,117 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-25 09:55:53,118 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x681750df to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,123 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e372785, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,124 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,126 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,127 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:55:53,129 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:55:53,130 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x02df63a4 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,134 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@501c5a88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,134 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,138 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,138 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,145 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,145 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,150 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,150 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,153 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,153 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,158 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,159 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,167 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,167 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,171 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,171 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,177 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,177 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:55:53,182 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:55:53,192 DEBUG [hconnection-0x4a3ed08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,193 DEBUG [hconnection-0x1e590adf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,193 DEBUG [hconnection-0x6c499665-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,194 DEBUG [hconnection-0x35805782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,194 DEBUG [hconnection-0x130eb58c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,194 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,195 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,196 DEBUG [hconnection-0x62f60509-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,201 DEBUG [hconnection-0x7dc2962e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,201 DEBUG [hconnection-0x406eb0ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,202 DEBUG [hconnection-0x45267bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,203 DEBUG [hconnection-0x26c68fae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:55:53,208 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,208 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,208 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,208 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,208 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,213 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,213 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:55:53,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:53,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:53,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:53,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:55:53,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:53,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:53,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:53,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:53,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:53,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:53,231 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:53,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:53,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:53,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,234 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:53,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:53,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416613246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416613246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416553244, max timestamp=1682416553245Num uniques 859; 2023-04-25 09:55:53,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:53,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416553244, max timestamp=1682416553245Num uniques 859; 2023-04-25 09:55:53,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:53,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416553244, max timestamp=1682416553245Num uniques 859; 2023-04-25 09:55:53,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:53,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416553254, max timestamp=1682416553256Num uniques 1018; 2023-04-25 09:55:53,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:55:53,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416553254, max timestamp=1682416553256Num uniques 1018; 2023-04-25 09:55:53,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416553254, max timestamp=1682416553256Num uniques 1018; 2023-04-25 09:55:53,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:55:53,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:55:53,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:53,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:53,261 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:53,262 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,263 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68026f6b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:53,266 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:53,266 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,267 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:53,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:53,268 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,269 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:53,269 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,269 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416613270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416613271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:53,272 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:53,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:53,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:53,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:53,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416613272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,272 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:53,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/ee7a1bd0e39946b2a298843c0e2b2b41 2023-04-25 09:55:53,274 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,277 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,281 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,281 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,281 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:53,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:53,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,282 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:53,282 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:53,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/38e21ab765214114952a95381d2b1bf7 2023-04-25 09:55:53,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/d20c015660da4d35ae165a46e42a12f8 2023-04-25 09:55:53,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/ee7a1bd0e39946b2a298843c0e2b2b41 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ee7a1bd0e39946b2a298843c0e2b2b41 2023-04-25 09:55:53,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ee7a1bd0e39946b2a298843c0e2b2b41, entries=250, sequenceid=12, filesize=16.2 K 2023-04-25 09:55:53,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/38e21ab765214114952a95381d2b1bf7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/38e21ab765214114952a95381d2b1bf7 2023-04-25 09:55:53,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/38e21ab765214114952a95381d2b1bf7, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 09:55:53,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/d20c015660da4d35ae165a46e42a12f8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d20c015660da4d35ae165a46e42a12f8 2023-04-25 09:55:53,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d20c015660da4d35ae165a46e42a12f8, entries=150, sequenceid=12, filesize=11.6 K 2023-04-25 09:55:53,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 6b4247a1e1c946a4543dfbc0edb52631 in 99ms, sequenceid=12, compaction requested=false 2023-04-25 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-25 09:55:53,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:53,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:53,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-25 09:55:53,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:53,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:53,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:53,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:53,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:53,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:53,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416613366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416613370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416613372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416613372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416613373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,382 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:53,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:53,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:53,392 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:53,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:53,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:53,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,393 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,393 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:53,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:53,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:53,404 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,404 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:53,404 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:53,404 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:53,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,406 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3161c667[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:53,407 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:53,407 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,409 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:53,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,410 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:53,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,411 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:53,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:53,413 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:53,413 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:53,413 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:53,414 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:53,414 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:53,416 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,416 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,416 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:53,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:53,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:53,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,420 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:53,420 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:53,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416613472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416613476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,520 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:53,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:53,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:53,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:53,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:53,524 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:53,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:53,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:53,524 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,525 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,526 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:53,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,528 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:53,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:53,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:53,537 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,537 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:53,537 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:53,537 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:53,540 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68dfb154[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:53,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:53,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,543 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:53,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:53,544 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:53,544 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:53,545 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,545 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:53,545 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,545 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:53,549 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:53,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:53,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:53,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:53,549 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:53,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,556 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,556 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:53,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:53,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,567 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:53,567 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:53,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:53,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416613580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416613581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416613581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,667 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:53,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:53,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:53,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:53,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:53,672 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:53,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:53,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:53,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,674 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:53,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:53,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:53,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:53,677 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:53,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416613677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc8c274[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:53,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:53,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:53,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:53,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:53,684 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416613686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:53,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:53,688 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:53,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:53,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:53,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:53,688 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:53,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:53,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:53,695 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:53,695 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:53,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:53,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/bc731e713d844002bb3c95b39b59213e 2023-04-25 09:55:53,795 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:53,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:53,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:53,798 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:53,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:53,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:53,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,806 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:53,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,808 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:53,809 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,809 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:53,809 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:53,809 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:53,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ca9d51[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:53,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:53,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:53,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:53,816 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:53,819 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:53,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:53,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:53,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:53,819 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:53,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:53,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:53,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:53,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,825 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:53,826 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:53,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416613887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416613889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416613891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,926 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:53,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:53,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:53,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:53,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:53,929 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:53,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:53,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:53,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,930 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,931 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:53,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:53,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:53,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:53,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:53,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:53,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa72fd1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:53,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:53,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:53,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:53,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:53,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:53,943 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:53,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:53,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:53,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:53,943 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:53,944 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,944 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:53,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:53,946 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:53,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:53,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:53,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:53,952 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:53,952 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:53,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:53,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:53,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:53,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416613979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:53,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416613987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,052 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:54,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:54,056 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:54,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:54,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:54,056 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,058 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:54,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:54,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:54,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:54,060 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:54,060 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:54,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,062 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:54,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a677260[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:54,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:54,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,064 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:54,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:54,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:54,070 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:54,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:54,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:54,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:54,070 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:54,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,074 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,075 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,075 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:54,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:54,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:54,077 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:54,077 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:54,177 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:54,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:54,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:54,180 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:54,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:54,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:54,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/433d61aacea047ba8a20807228f54d74 2023-04-25 09:55:54,182 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,182 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,182 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,182 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:54,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:54,184 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,184 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:54,184 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:54,184 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:54,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1771c03a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:54,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:54,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,192 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,192 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:54,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:54,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:54,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:54,197 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:54,197 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:54,197 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:54,199 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,199 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,200 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:54,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:54,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:54,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,204 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:54,204 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:54,204 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,204 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,204 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,204 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,304 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:54,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:54,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:54,307 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:54,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:54,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:54,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,308 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:54,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:54,311 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,311 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:54,311 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:54,311 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:54,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f5e790[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:54,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:54,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,317 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:54,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,317 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:54,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:54,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:54,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:54,322 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:54,322 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:54,322 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:54,322 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:54,322 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:54,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,324 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,325 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:54,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:54,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:54,328 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:54,328 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:54,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416614393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416614395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416614396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,428 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:54,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:54,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:54,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:54,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:54,432 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:54,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:54,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:54,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,434 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,434 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:54,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:54,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:54,436 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,436 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:54,437 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:54,437 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:54,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:54,438 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,438 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:54,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45da9f98[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:54,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:54,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:54,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:54,444 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:54,447 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:54,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:54,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:54,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:54,447 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:54,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,450 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,451 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,452 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,455 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:54,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:54,455 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:54,456 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,456 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416614482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:54,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416614496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,556 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:54,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:54,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:54,560 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:54,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:54,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:54,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:54,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:54,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:54,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:54,563 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:54,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:54,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:54,565 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,566 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,566 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244f0131[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:54,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:54,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:54,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:54,573 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,573 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:54,573 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,573 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:54,575 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:54,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:54,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:54,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:54,576 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:54,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:54,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:54,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,580 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,582 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:54,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:54,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,582 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:54,582 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:54,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:54,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/4d1ed629120541a0a31104e033ec21c8 2023-04-25 09:55:54,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/bc731e713d844002bb3c95b39b59213e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/bc731e713d844002bb3c95b39b59213e 2023-04-25 09:55:54,599 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/bc731e713d844002bb3c95b39b59213e, entries=200, sequenceid=66, filesize=13.9 K 2023-04-25 09:55:54,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/433d61aacea047ba8a20807228f54d74 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/433d61aacea047ba8a20807228f54d74 2023-04-25 09:55:54,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/433d61aacea047ba8a20807228f54d74, entries=150, sequenceid=66, filesize=11.6 K 2023-04-25 09:55:54,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/4d1ed629120541a0a31104e033ec21c8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4d1ed629120541a0a31104e033ec21c8 2023-04-25 09:55:54,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4d1ed629120541a0a31104e033ec21c8, entries=150, sequenceid=66, filesize=11.6 K 2023-04-25 09:55:54,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for 6b4247a1e1c946a4543dfbc0edb52631 in 1254ms, sequenceid=66, compaction requested=false 2023-04-25 09:55:54,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:54,682 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:54,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:54,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:54,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:54,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:54,686 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:54,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:54,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:54,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,689 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:54,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:54,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:54,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:54,691 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:54,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:54,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:54,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:54,727 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/01e9bd13eacc4bc39a84465f516eb6a3 2023-04-25 09:55:54,741 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/b7322c1c2a5c451d95ecf223277b38ac 2023-04-25 09:55:55,151 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3498732266c94a498fd5f2eef45b1415 2023-04-25 09:55:55,155 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/01e9bd13eacc4bc39a84465f516eb6a3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/01e9bd13eacc4bc39a84465f516eb6a3 2023-04-25 09:55:55,159 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/01e9bd13eacc4bc39a84465f516eb6a3, entries=150, sequenceid=75, filesize=11.6 K 2023-04-25 09:55:55,159 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/b7322c1c2a5c451d95ecf223277b38ac as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/b7322c1c2a5c451d95ecf223277b38ac 2023-04-25 09:55:55,163 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/b7322c1c2a5c451d95ecf223277b38ac, entries=150, sequenceid=75, filesize=11.6 K 2023-04-25 09:55:55,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3498732266c94a498fd5f2eef45b1415 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3498732266c94a498fd5f2eef45b1415 2023-04-25 09:55:55,167 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3498732266c94a498fd5f2eef45b1415, entries=150, sequenceid=75, filesize=11.6 K 2023-04-25 09:55:55,168 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 6b4247a1e1c946a4543dfbc0edb52631 in 477ms, sequenceid=75, compaction requested=true 2023-04-25 09:55:55,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:55,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d267592[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:55,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:55,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:55,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:55,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:55,176 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:55,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:55,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:55,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:55,177 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:55,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,185 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:55,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:55,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,186 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:55,186 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:55,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:55,236 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 09:55:55,286 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:55,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:55,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:55,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:55,289 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:55,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:55,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:55,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,290 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,291 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:55,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:55,292 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,292 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:55,293 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2173ccc1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:55,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:55,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,307 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,307 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:55,307 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:55,307 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:55,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:55,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:55,309 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:55,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:55,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:55,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:55,309 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:55,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,316 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:55,316 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:55,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:55,317 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:55,317 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:55,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:55,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:55,417 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:55,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:55,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:55,420 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:55,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:55,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:55,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/66c75fb9e7d94bc2a1ccc6cdab1145f8 2023-04-25 09:55:55,423 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,423 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,423 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:55,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:55,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:55,425 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,425 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:55,425 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:55,425 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,426 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3044b128[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:55,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:55,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,430 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:55,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:55,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/577f3eca1bda433099da2c981adc776e 2023-04-25 09:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:55,435 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:55,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:55,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:55,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:55,435 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:55,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,444 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,446 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:55,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,446 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:55,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,446 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:55,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416555440, max timestamp=1682416555466Num uniques 859; 2023-04-25 09:55:55,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:55:55,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416555440, max timestamp=1682416555466Num uniques 859; 2023-04-25 09:55:55,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:55:55,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416555440, max timestamp=1682416555466Num uniques 859; 2023-04-25 09:55:55,469 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:55:55,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416615499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416615501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416615502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416615507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416615507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,546 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:55,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:55,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:55,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:55,549 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:55,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:55,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:55,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,551 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:55,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:55,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:55,552 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:55,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:55,553 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,555 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cde7898[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:55,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:55,557 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:55,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:55,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:55,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:55,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:55,561 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:55,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:55,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:55,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:55,562 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:55,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,564 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:55,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:55,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,569 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:55,569 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:55,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416615608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416615612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416615615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416615615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,669 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:55,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:55,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:55,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:55,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:55,672 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:55,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:55,672 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:55,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,676 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:55,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:55,678 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,678 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:55,678 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:55,678 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62da0a74[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:55,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:55,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:55,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:55,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:55,688 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:55,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:55,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:55,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:55,689 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:55,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,694 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,694 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:55,694 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:55,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,795 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:55,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:55,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:55,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:55,798 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:55,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:55,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:55,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,799 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:55,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:55,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:55,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:55,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:55,802 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:55,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d70d72a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:55,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:55,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:55,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:55,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:55,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:55,815 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:55,818 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:55,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:55,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:55,818 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:55,818 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:55,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416615816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416615820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416615822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416615823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,830 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:55,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:55,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:55,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:55,830 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:55,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:55,830 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:55,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/750854fcd28d4fd686ca449f00100f42 2023-04-25 09:55:55,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/66c75fb9e7d94bc2a1ccc6cdab1145f8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/66c75fb9e7d94bc2a1ccc6cdab1145f8 2023-04-25 09:55:55,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/66c75fb9e7d94bc2a1ccc6cdab1145f8, entries=200, sequenceid=86, filesize=13.9 K 2023-04-25 09:55:55,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/577f3eca1bda433099da2c981adc776e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/577f3eca1bda433099da2c981adc776e 2023-04-25 09:55:55,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/577f3eca1bda433099da2c981adc776e, entries=150, sequenceid=86, filesize=11.6 K 2023-04-25 09:55:55,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/750854fcd28d4fd686ca449f00100f42 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/750854fcd28d4fd686ca449f00100f42 2023-04-25 09:55:55,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/750854fcd28d4fd686ca449f00100f42, entries=150, sequenceid=86, filesize=11.6 K 2023-04-25 09:55:55,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 6b4247a1e1c946a4543dfbc0edb52631 in 451ms, sequenceid=86, compaction requested=true 2023-04-25 09:55:55,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:55,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:55,863 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:55,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:55,863 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:55:55,863 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:55,864 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56904 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:55,864 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:55:55,864 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:55,864 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,864 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:55:55,864 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ee7a1bd0e39946b2a298843c0e2b2b41, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/bc731e713d844002bb3c95b39b59213e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/01e9bd13eacc4bc39a84465f516eb6a3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/66c75fb9e7d94bc2a1ccc6cdab1145f8] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=55.6 K 2023-04-25 09:55:55,864 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,865 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/38e21ab765214114952a95381d2b1bf7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/433d61aacea047ba8a20807228f54d74, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/b7322c1c2a5c451d95ecf223277b38ac, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/577f3eca1bda433099da2c981adc776e] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=46.4 K 2023-04-25 09:55:55,865 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ee7a1bd0e39946b2a298843c0e2b2b41, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682416553226 2023-04-25 09:55:55,865 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 38e21ab765214114952a95381d2b1bf7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682416553226 2023-04-25 09:55:55,865 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting bc731e713d844002bb3c95b39b59213e, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682416553269 2023-04-25 09:55:55,865 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 433d61aacea047ba8a20807228f54d74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682416553269 2023-04-25 09:55:55,865 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 01e9bd13eacc4bc39a84465f516eb6a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682416553360 2023-04-25 09:55:55,866 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b7322c1c2a5c451d95ecf223277b38ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682416553360 2023-04-25 09:55:55,866 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 66c75fb9e7d94bc2a1ccc6cdab1145f8, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682416555403 2023-04-25 09:55:55,866 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 577f3eca1bda433099da2c981adc776e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682416555403 2023-04-25 09:55:55,874 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:55,877 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:55,885 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2e99ed48df8b47c99c505810510b9712 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2e99ed48df8b47c99c505810510b9712 2023-04-25 09:55:55,890 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 2e99ed48df8b47c99c505810510b9712(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:55,890 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:55,890 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=12, startTime=1682416555863; duration=0sec 2023-04-25 09:55:55,890 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:55,890 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:55,892 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:55,892 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:55:55,892 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,892 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d20c015660da4d35ae165a46e42a12f8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4d1ed629120541a0a31104e033ec21c8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3498732266c94a498fd5f2eef45b1415, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/750854fcd28d4fd686ca449f00100f42] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=46.4 K 2023-04-25 09:55:55,893 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting d20c015660da4d35ae165a46e42a12f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682416553226 2023-04-25 09:55:55,893 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4d1ed629120541a0a31104e033ec21c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1682416553269 2023-04-25 09:55:55,893 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3498732266c94a498fd5f2eef45b1415, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1682416553360 2023-04-25 09:55:55,894 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 750854fcd28d4fd686ca449f00100f42, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682416555403 2023-04-25 09:55:55,900 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:55,931 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:55,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:55,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:55,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:55,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:55,934 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:55,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:55,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:55,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:55,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:55,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:55,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:55,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:55,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:55,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:55,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:55,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:55,939 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:55,939 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:55,939 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-25 09:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:55,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:55,950 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/df75588bbb9e43eebf587ec733adf085 2023-04-25 09:55:55,965 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/0865fe1820164363853b1a844200ecd0 2023-04-25 09:55:55,978 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/fd91c098a2c24520abc807be736f0a5b 2023-04-25 09:55:55,982 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/df75588bbb9e43eebf587ec733adf085 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df75588bbb9e43eebf587ec733adf085 2023-04-25 09:55:55,986 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df75588bbb9e43eebf587ec733adf085, entries=150, sequenceid=123, filesize=11.6 K 2023-04-25 09:55:55,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/0865fe1820164363853b1a844200ecd0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0865fe1820164363853b1a844200ecd0 2023-04-25 09:55:55,989 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0865fe1820164363853b1a844200ecd0, entries=150, sequenceid=123, filesize=11.6 K 2023-04-25 09:55:55,990 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/fd91c098a2c24520abc807be736f0a5b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fd91c098a2c24520abc807be736f0a5b 2023-04-25 09:55:55,995 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fd91c098a2c24520abc807be736f0a5b, entries=150, sequenceid=123, filesize=11.6 K 2023-04-25 09:55:55,996 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=0 B/0 for 6b4247a1e1c946a4543dfbc0edb52631 in 57ms, sequenceid=123, compaction requested=false 2023-04-25 09:55:55,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:55,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:55,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:55,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a8f05d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,001 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,003 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,003 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,003 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,006 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,006 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,010 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,010 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,010 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,011 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,012 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,014 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,015 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,018 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,018 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,019 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,019 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,118 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,122 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,122 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,125 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,126 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,126 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,126 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,127 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,127 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,128 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-25 09:55:56,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:56,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:56,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:56,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:56,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:56,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:56,139 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/df9eb23168e34148adc7131a4b378045 2023-04-25 09:55:56,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:56,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,164 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/266c1dc680064e089271004ab6711c77 2023-04-25 09:55:56,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416556152, max timestamp=1682416556171Num uniques 859; 2023-04-25 09:55:56,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:56,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416556152, max timestamp=1682416556171Num uniques 859; 2023-04-25 09:55:56,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:56,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416556152, max timestamp=1682416556171Num uniques 859; 2023-04-25 09:55:56,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:55:56,178 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/9dd88a6fdc0c422db0bdf7bcf1e25c82 2023-04-25 09:55:56,182 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/df9eb23168e34148adc7131a4b378045 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df9eb23168e34148adc7131a4b378045 2023-04-25 09:55:56,186 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df9eb23168e34148adc7131a4b378045, entries=50, sequenceid=127, filesize=7.0 K 2023-04-25 09:55:56,187 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/266c1dc680064e089271004ab6711c77 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/266c1dc680064e089271004ab6711c77 2023-04-25 09:55:56,191 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/266c1dc680064e089271004ab6711c77, entries=50, sequenceid=127, filesize=7.0 K 2023-04-25 09:55:56,194 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/9dd88a6fdc0c422db0bdf7bcf1e25c82 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/9dd88a6fdc0c422db0bdf7bcf1e25c82 2023-04-25 09:55:56,198 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/9dd88a6fdc0c422db0bdf7bcf1e25c82, entries=50, sequenceid=127, filesize=7.0 K 2023-04-25 09:55:56,199 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=116.29 KB/119080 for 6b4247a1e1c946a4543dfbc0edb52631 in 71ms, sequenceid=127, compaction requested=true 2023-04-25 09:55:56,199 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:56,199 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57ad4b68[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,205 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=123.00 KB heapSize=345.53 KB 2023-04-25 09:55:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:56,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:56,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:56,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:56,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:56,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,215 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/aab3846a50ba4866a795f328e551f9ca 2023-04-25 09:55:56,221 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,225 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,224 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,225 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416616222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416616223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416616224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416616236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,295 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/0860af2586474602b5ff68031415ec51 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0860af2586474602b5ff68031415ec51 2023-04-25 09:55:56,299 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 0860af2586474602b5ff68031415ec51(size=11.7 K), total size for store is 30.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:56,299 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:56,299 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=12, startTime=1682416555863; duration=0sec 2023-04-25 09:55:56,299 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:56,310 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/912fd657c28145ddbdd31bb61130fb4d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/912fd657c28145ddbdd31bb61130fb4d 2023-04-25 09:55:56,314 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 912fd657c28145ddbdd31bb61130fb4d(size=11.7 K), total size for store is 30.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:56,314 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:56,314 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=12, startTime=1682416555863; duration=0sec 2023-04-25 09:55:56,314 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:56,325 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,328 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,328 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,328 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,328 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,330 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,330 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,344 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416616339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416616340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,346 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416616337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416616340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65967243[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,362 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,362 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,362 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,362 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,365 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,365 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,366 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,371 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,374 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,374 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,474 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,478 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,487 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,487 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,487 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,487 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,488 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e04305c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,491 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,491 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,498 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,498 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,502 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,504 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,506 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,506 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,506 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,506 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:56,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416616547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416616547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416616548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416616549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,607 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,610 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,610 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,610 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,610 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,612 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,613 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,614 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,615 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,615 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,615 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,615 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,617 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31a105d9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,619 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,619 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,620 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,620 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,620 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,621 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,622 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,624 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,624 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,624 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,624 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,624 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,626 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,627 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/6079a4a9f2944ecf8f46050ae59d0d26 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,632 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,633 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,633 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:56,733 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,736 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,741 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ceaee8e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,743 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,743 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,744 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,744 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,746 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,749 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,749 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,749 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,750 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,752 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:56,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,754 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,755 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,755 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,855 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416616851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,858 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,859 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,859 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416616852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,860 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,862 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,862 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,862 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,862 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416616854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:56,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416616856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,871 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,871 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,871 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,872 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a38813a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,873 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,873 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,874 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:56,874 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:56,878 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:56,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:56,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:56,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:56,878 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:56,879 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,879 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,879 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,880 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,882 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,886 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:56,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:56,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,886 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:56,886 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,886 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:56,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:56,986 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:56,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:56,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:56,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:56,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:56,989 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:56,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:56,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:56,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:56,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:56,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:56,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:56,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:56,993 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,993 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:56,993 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:56,993 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:56,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a1ccf1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:56,998 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:56,998 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:56,999 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:56,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:56,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,001 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:57,003 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:57,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:57,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:57,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:57,003 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,005 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,008 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:57,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:57,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,009 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:57,009 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:57,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/c80b413f3da448d5be1477b46baaa302 2023-04-25 09:55:57,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/aab3846a50ba4866a795f328e551f9ca as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/aab3846a50ba4866a795f328e551f9ca 2023-04-25 09:55:57,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/aab3846a50ba4866a795f328e551f9ca, entries=200, sequenceid=162, filesize=14.1 K 2023-04-25 09:55:57,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/6079a4a9f2944ecf8f46050ae59d0d26 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6079a4a9f2944ecf8f46050ae59d0d26 2023-04-25 09:55:57,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6079a4a9f2944ecf8f46050ae59d0d26, entries=150, sequenceid=162, filesize=11.8 K 2023-04-25 09:55:57,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/c80b413f3da448d5be1477b46baaa302 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/c80b413f3da448d5be1477b46baaa302 2023-04-25 09:55:57,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/c80b413f3da448d5be1477b46baaa302, entries=150, sequenceid=162, filesize=11.8 K 2023-04-25 09:55:57,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~365.41 KB/374176, currentSize=53.67 KB/54960 for 6b4247a1e1c946a4543dfbc0edb52631 in 884ms, sequenceid=162, compaction requested=true 2023-04-25 09:55:57,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:57,091 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:57,091 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:57,092 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45536 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:57,092 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43146 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:57,092 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:55:57,092 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:55:57,092 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,092 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,092 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2e99ed48df8b47c99c505810510b9712, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df75588bbb9e43eebf587ec733adf085, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df9eb23168e34148adc7131a4b378045, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/aab3846a50ba4866a795f328e551f9ca] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=44.5 K 2023-04-25 09:55:57,092 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0860af2586474602b5ff68031415ec51, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0865fe1820164363853b1a844200ecd0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/266c1dc680064e089271004ab6711c77, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6079a4a9f2944ecf8f46050ae59d0d26] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=42.1 K 2023-04-25 09:55:57,093 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2e99ed48df8b47c99c505810510b9712, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682416555403 2023-04-25 09:55:57,093 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0860af2586474602b5ff68031415ec51, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682416555403 2023-04-25 09:55:57,093 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting df75588bbb9e43eebf587ec733adf085, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682416555497 2023-04-25 09:55:57,095 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0865fe1820164363853b1a844200ecd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682416555497 2023-04-25 09:55:57,095 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting df9eb23168e34148adc7131a4b378045, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416556123 2023-04-25 09:55:57,095 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 266c1dc680064e089271004ab6711c77, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416556123 2023-04-25 09:55:57,095 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6079a4a9f2944ecf8f46050ae59d0d26, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682416556188 2023-04-25 09:55:57,095 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting aab3846a50ba4866a795f328e551f9ca, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682416556188 2023-04-25 09:55:57,103 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:57,109 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:57,109 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:57,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:57,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:57,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:57,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:57,140 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:57,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:57,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:57,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:57,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,145 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:57,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:57,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:57,148 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,148 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:57,149 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:55:57,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:57,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:57,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:57,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:57,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:57,149 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:57,157 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/517c8d0d5e0e49fbbd284d4f7fee8e9a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/517c8d0d5e0e49fbbd284d4f7fee8e9a 2023-04-25 09:55:57,158 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/994f0a2a66594446b0aff4e7238e826d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/994f0a2a66594446b0aff4e7238e826d 2023-04-25 09:55:57,165 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 517c8d0d5e0e49fbbd284d4f7fee8e9a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:57,165 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:57,165 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=12, startTime=1682416557091; duration=0sec 2023-04-25 09:55:57,165 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 994f0a2a66594446b0aff4e7238e826d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:57,165 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:57,165 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:57,165 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:55:57,165 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=12, startTime=1682416557091; duration=0sec 2023-04-25 09:55:57,165 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:57,167 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43146 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:55:57,167 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:55:57,167 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,167 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/912fd657c28145ddbdd31bb61130fb4d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fd91c098a2c24520abc807be736f0a5b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/9dd88a6fdc0c422db0bdf7bcf1e25c82, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/c80b413f3da448d5be1477b46baaa302] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=42.1 K 2023-04-25 09:55:57,167 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 912fd657c28145ddbdd31bb61130fb4d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682416555403 2023-04-25 09:55:57,168 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting fd91c098a2c24520abc807be736f0a5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1682416555497 2023-04-25 09:55:57,168 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9dd88a6fdc0c422db0bdf7bcf1e25c82, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1682416556123 2023-04-25 09:55:57,168 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c80b413f3da448d5be1477b46baaa302, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682416556188 2023-04-25 09:55:57,172 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/98c7027bb22d4d399154e4b06e7101e0 2023-04-25 09:55:57,177 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#441 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:57,192 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/6bbaf1f0d5bb4e38829ca819d44008c6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/6bbaf1f0d5bb4e38829ca819d44008c6 2023-04-25 09:55:57,194 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/ee3ebec3547746f398e29bfaf43da5e4 2023-04-25 09:55:57,197 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 6bbaf1f0d5bb4e38829ca819d44008c6(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:57,197 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:57,198 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=12, startTime=1682416557091; duration=0sec 2023-04-25 09:55:57,198 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:57,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:57,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:57,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416557371, max timestamp=1682416557387Num uniques 859; 2023-04-25 09:55:57,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:55:57,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416557371, max timestamp=1682416557387Num uniques 859; 2023-04-25 09:55:57,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:55:57,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416557371, max timestamp=1682416557387Num uniques 859; 2023-04-25 09:55:57,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:55:57,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416617406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416617407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416617409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416617410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416617513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416617513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416617514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416617516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,521 DEBUG [Thread-3205] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:55:57,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416617517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,614 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/73648a6c0f7c498ca57f5949ec9296b6 2023-04-25 09:55:57,618 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/98c7027bb22d4d399154e4b06e7101e0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/98c7027bb22d4d399154e4b06e7101e0 2023-04-25 09:55:57,621 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/98c7027bb22d4d399154e4b06e7101e0, entries=150, sequenceid=175, filesize=11.8 K 2023-04-25 09:55:57,622 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/ee3ebec3547746f398e29bfaf43da5e4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ee3ebec3547746f398e29bfaf43da5e4 2023-04-25 09:55:57,625 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ee3ebec3547746f398e29bfaf43da5e4, entries=150, sequenceid=175, filesize=11.8 K 2023-04-25 09:55:57,626 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/73648a6c0f7c498ca57f5949ec9296b6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/73648a6c0f7c498ca57f5949ec9296b6 2023-04-25 09:55:57,630 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/73648a6c0f7c498ca57f5949ec9296b6, entries=150, sequenceid=175, filesize=11.8 K 2023-04-25 09:55:57,631 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 6b4247a1e1c946a4543dfbc0edb52631 in 482ms, sequenceid=175, compaction requested=false 2023-04-25 09:55:57,631 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:57,631 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,631 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:57,631 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:57,636 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c979651[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:57,638 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:57,638 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,639 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:57,639 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,639 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:57,640 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,641 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:57,643 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:57,643 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:57,643 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:57,643 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:57,643 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:57,644 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,644 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,644 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,645 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,647 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:57,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,650 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,650 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,650 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:57,651 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:57,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:57,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-25 09:55:57,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:57,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:57,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:57,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:57,726 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:57,726 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:57,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2d7eaba66c834cefb4daea1e95206bcf 2023-04-25 09:55:57,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/9ee3178729cd4be48c6fc72521bb248e 2023-04-25 09:55:57,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416617734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416617735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416617736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416617737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,751 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:57,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:57,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:57,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:57,754 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:57,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:57,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:57,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:57,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,754 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,755 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,756 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,756 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:57,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:57,757 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,757 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:57,757 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:57,757 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:57,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:57,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@275299d3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:57,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:57,761 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,762 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:57,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:57,766 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:57,769 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:57,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:57,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:57,769 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:57,769 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:57,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,773 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,776 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:57,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:57,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,777 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,777 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,777 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:57,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,777 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,777 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416617843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416617844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416617848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:57,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416617848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,877 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:57,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:57,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:57,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:57,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:57,880 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:57,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:57,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:57,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:57,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,881 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,882 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,882 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:57,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:57,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:57,884 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:57,884 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:57,884 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:57,885 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:57,885 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:57,885 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,885 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:57,885 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:57,885 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:57,886 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,888 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397dfd52[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:57,890 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:57,890 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,891 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,891 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:57,891 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:57,891 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:57,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:57,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:57,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:57,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:57,894 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:57,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:57,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:57,894 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:57,894 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:57,895 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,895 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,896 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:57,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,896 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:57,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:57,904 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:57,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:57,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:57,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:57,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:57,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:57,904 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:57,905 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:58,005 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:58,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:58,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:58,008 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:58,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:58,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:58,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:58,008 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,017 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:58,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:58,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:58,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:58,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:58,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:58,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:58,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:58,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:58,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:58,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@727078c8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:58,024 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:58,024 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:58,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:58,034 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:58,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:58,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:58,037 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:58,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:58,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:58,037 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:58,037 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:58,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:58,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:58,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:58,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:58,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:58,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:58,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:58,044 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:58,044 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:58,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:58,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416618048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416618049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416618052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416618052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,145 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:58,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:58,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:58,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:58,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:58,148 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:58,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:58,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:58,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:58,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,150 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:58,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:58,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:58,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:58,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:58,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:58,152 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:58,152 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:58,152 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:58,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:58,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/0dbcee85358b4934b604df7226a8cc71 2023-04-25 09:55:58,155 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:58,155 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:58,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:58,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32955dc9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:58,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:58,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2d7eaba66c834cefb4daea1e95206bcf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2d7eaba66c834cefb4daea1e95206bcf 2023-04-25 09:55:58,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:58,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:58,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2d7eaba66c834cefb4daea1e95206bcf, entries=200, sequenceid=213, filesize=14.1 K 2023-04-25 09:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/9ee3178729cd4be48c6fc72521bb248e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9ee3178729cd4be48c6fc72521bb248e 2023-04-25 09:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:58,163 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:58,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:58,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:58,163 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:58,163 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:58,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9ee3178729cd4be48c6fc72521bb248e, entries=150, sequenceid=213, filesize=11.8 K 2023-04-25 09:55:58,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/0dbcee85358b4934b604df7226a8cc71 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/0dbcee85358b4934b604df7226a8cc71 2023-04-25 09:55:58,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:58,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:58,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:58,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/0dbcee85358b4934b604df7226a8cc71, entries=150, sequenceid=213, filesize=11.8 K 2023-04-25 09:55:58,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 6b4247a1e1c946a4543dfbc0edb52631 in 448ms, sequenceid=213, compaction requested=true 2023-04-25 09:55:58,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,174 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:58,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:58,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:58,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:55:58,174 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:58,177 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38772 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:58,177 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:58,177 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:55:58,177 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:55:58,177 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,177 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,177 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/517c8d0d5e0e49fbbd284d4f7fee8e9a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/98c7027bb22d4d399154e4b06e7101e0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2d7eaba66c834cefb4daea1e95206bcf] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=37.9 K 2023-04-25 09:55:58,177 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/994f0a2a66594446b0aff4e7238e826d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ee3ebec3547746f398e29bfaf43da5e4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9ee3178729cd4be48c6fc72521bb248e] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=35.5 K 2023-04-25 09:55:58,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:58,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:58,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:58,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:58,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:58,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:58,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:58,178 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 994f0a2a66594446b0aff4e7238e826d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682416556188 2023-04-25 09:55:58,178 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 517c8d0d5e0e49fbbd284d4f7fee8e9a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682416556188 2023-04-25 09:55:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:58,178 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:58,178 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:58,178 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ee3ebec3547746f398e29bfaf43da5e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1682416556220 2023-04-25 09:55:58,179 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 98c7027bb22d4d399154e4b06e7101e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1682416556220 2023-04-25 09:55:58,179 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9ee3178729cd4be48c6fc72521bb248e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682416557405 2023-04-25 09:55:58,179 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2d7eaba66c834cefb4daea1e95206bcf, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682416557405 2023-04-25 09:55:58,186 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:58,187 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#448 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:58,211 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/53aecdeecf5c47689b05ac89c98c749f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/53aecdeecf5c47689b05ac89c98c749f 2023-04-25 09:55:58,214 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 53aecdeecf5c47689b05ac89c98c749f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:58,214 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:58,214 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=13, startTime=1682416558174; duration=0sec 2023-04-25 09:55:58,214 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:55:58,215 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:55:58,215 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:55:58,215 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:55:58,218 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,218 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/6bbaf1f0d5bb4e38829ca819d44008c6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/73648a6c0f7c498ca57f5949ec9296b6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/0dbcee85358b4934b604df7226a8cc71] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=35.5 K 2023-04-25 09:55:58,218 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 6bbaf1f0d5bb4e38829ca819d44008c6, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1682416556188 2023-04-25 09:55:58,218 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 73648a6c0f7c498ca57f5949ec9296b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1682416556220 2023-04-25 09:55:58,219 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0dbcee85358b4934b604df7226a8cc71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682416557405 2023-04-25 09:55:58,235 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:55:58,245 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/5e44fb879bce4660bf0061e313a54f46 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/5e44fb879bce4660bf0061e313a54f46 2023-04-25 09:55:58,250 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 5e44fb879bce4660bf0061e313a54f46(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:58,250 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:58,250 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=13, startTime=1682416558174; duration=0sec 2023-04-25 09:55:58,250 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:58,278 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:58,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:58,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:58,282 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:58,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:58,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:58,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:58,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,284 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:58,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:58,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:58,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:58,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:58,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:58,286 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:58,286 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:58,286 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:58,298 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/b8e8b42ab2004cd7868f529c850d1b6c 2023-04-25 09:55:58,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:58,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:58,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682416558366, max timestamp=1682416558406Num uniques 859; 2023-04-25 09:55:58,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1682416558366, max timestamp=1682416558406Num uniques 859; 2023-04-25 09:55:58,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1682416558366, max timestamp=1682416558406Num uniques 859; 2023-04-25 09:55:58,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:58,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:55:58,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=8378, suffix heap size=117064,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=183 2023-04-25 09:55:58,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416618444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416618445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416618446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416618448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416618555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416618555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416618555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416618555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,602 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/cdee0da979254bab955e5b55945651bd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/cdee0da979254bab955e5b55945651bd 2023-04-25 09:55:58,606 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into cdee0da979254bab955e5b55945651bd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:55:58,606 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:58,606 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=13, startTime=1682416558174; duration=0sec 2023-04-25 09:55:58,606 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:55:58,710 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/86581cf452854997ab3c35e97c6a97d0 2023-04-25 09:55:58,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416618760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416618760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416618761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:58,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:58,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416618762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416619068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416619068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416619069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416619069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,121 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/d7a0e7aa72fc434ea74092c6bc297a56 2023-04-25 09:55:59,124 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/b8e8b42ab2004cd7868f529c850d1b6c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b8e8b42ab2004cd7868f529c850d1b6c 2023-04-25 09:55:59,128 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b8e8b42ab2004cd7868f529c850d1b6c, entries=150, sequenceid=225, filesize=11.8 K 2023-04-25 09:55:59,128 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/86581cf452854997ab3c35e97c6a97d0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/86581cf452854997ab3c35e97c6a97d0 2023-04-25 09:55:59,132 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/86581cf452854997ab3c35e97c6a97d0, entries=150, sequenceid=225, filesize=11.8 K 2023-04-25 09:55:59,132 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/d7a0e7aa72fc434ea74092c6bc297a56 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d7a0e7aa72fc434ea74092c6bc297a56 2023-04-25 09:55:59,135 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d7a0e7aa72fc434ea74092c6bc297a56, entries=150, sequenceid=225, filesize=11.8 K 2023-04-25 09:55:59,136 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=137.49 KB/140794 for 6b4247a1e1c946a4543dfbc0edb52631 in 850ms, sequenceid=225, compaction requested=false 2023-04-25 09:55:59,136 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:59,136 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:59,136 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:59,139 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:59,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,139 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bac649[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:59,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:59,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,142 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,142 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:59,142 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:59,142 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:59,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,143 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:59,144 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:59,147 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:59,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:59,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:59,147 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:59,147 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:59,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:59,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:59,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,158 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:59,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:59,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:55:59,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:59,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:59,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:59,160 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:59,160 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:59,260 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:59,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:59,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:59,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:59,266 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:59,266 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:59,266 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:59,266 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:59,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,268 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,268 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:59,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:59,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,269 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:59,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:59,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:59,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:59,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:59,270 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:59,270 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=137.49 KB heapSize=387.81 KB 2023-04-25 09:55:59,270 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:59,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:59,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:59,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:59,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:59,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:55:59,281 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/b59368168cbd424a810c73fd245d5764 2023-04-25 09:55:59,293 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/6a7f07ef4fc04d34860bff301833cce8 2023-04-25 09:55:59,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:59,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:59,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416619585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416619586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416619591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416619592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416619693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416619693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416619693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416619698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,703 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=43.57 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/984efee93dc84de2b6139c526710eec3 2023-04-25 09:55:59,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/b59368168cbd424a810c73fd245d5764 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b59368168cbd424a810c73fd245d5764 2023-04-25 09:55:59,711 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b59368168cbd424a810c73fd245d5764, entries=150, sequenceid=262, filesize=11.9 K 2023-04-25 09:55:59,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/6a7f07ef4fc04d34860bff301833cce8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6a7f07ef4fc04d34860bff301833cce8 2023-04-25 09:55:59,714 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6a7f07ef4fc04d34860bff301833cce8, entries=150, sequenceid=262, filesize=11.9 K 2023-04-25 09:55:59,715 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/984efee93dc84de2b6139c526710eec3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/984efee93dc84de2b6139c526710eec3 2023-04-25 09:55:59,718 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/984efee93dc84de2b6139c526710eec3, entries=150, sequenceid=262, filesize=11.9 K 2023-04-25 09:55:59,718 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~137.49 KB/140794, heapSize ~387.77 KB/397072, currentSize=46.96 KB/48090 for 6b4247a1e1c946a4543dfbc0edb52631 in 448ms, sequenceid=262, compaction requested=true 2023-04-25 09:55:59,718 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:55:59,718 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:59,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:55:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:55:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:55:59,719 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:55:59,720 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:59,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:59,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57fb2c41[Count = 0] remaining members to acquire global barrier 2023-04-25 09:55:59,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:55:59,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,726 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:55:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:55:59,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:55:59,726 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,726 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:55:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:55:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:55:59,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:55:59,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:55:59,731 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:55:59,731 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:55:59,731 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:55:59,731 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:55:59,731 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:55:59,732 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,732 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:55:59,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,732 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:55:59,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:59,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:55:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,735 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:55:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:55:59,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:55:59,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:55:59,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,737 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:55:59,737 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:55:59,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:55:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:55:59,838 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:55:59,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:55:59,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:55:59,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:55:59,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:55:59,841 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:55:59,841 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:59,841 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:55:59,841 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:55:59,841 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,841 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,842 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,843 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:55:59,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:55:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:55:59,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,843 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:55:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:55:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:55:59,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:55:59,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:55:59,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:55:59,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:55:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:55:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:55:59,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:55:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:55:59,845 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:55:59,845 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:55:59,845 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:55:59,845 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:55:59,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:55:59,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:59,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:55:59,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:59,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:55:59,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:55:59,858 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/7ec38f999f614856858340ed15484f72 2023-04-25 09:55:59,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:55:59,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:55:59,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416559920, max timestamp=1682416559934Num uniques 859; 2023-04-25 09:55:59,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416559920, max timestamp=1682416559934Num uniques 859; 2023-04-25 09:55:59,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:55:59,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:55:59,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416559920, max timestamp=1682416559934Num uniques 859; 2023-04-25 09:55:59,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:55:59,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416619970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416619971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416619972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:55:59,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:55:59,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416619972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416620082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416620083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416620083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416620085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,270 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/353fc21a742c4547bcbc88cbf1b936c6 2023-04-25 09:56:00,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416620288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416620288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416620288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416620292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416620594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416620594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54328 deadline: 1682416620595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:00,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416620598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,681 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3530c8f9348241aeb8b920d85dfb2238 2023-04-25 09:56:00,685 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/7ec38f999f614856858340ed15484f72 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7ec38f999f614856858340ed15484f72 2023-04-25 09:56:00,688 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7ec38f999f614856858340ed15484f72, entries=100, sequenceid=272, filesize=9.5 K 2023-04-25 09:56:00,689 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/353fc21a742c4547bcbc88cbf1b936c6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/353fc21a742c4547bcbc88cbf1b936c6 2023-04-25 09:56:00,692 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/353fc21a742c4547bcbc88cbf1b936c6, entries=100, sequenceid=272, filesize=9.5 K 2023-04-25 09:56:00,693 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3530c8f9348241aeb8b920d85dfb2238 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3530c8f9348241aeb8b920d85dfb2238 2023-04-25 09:56:00,696 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3530c8f9348241aeb8b920d85dfb2238, entries=100, sequenceid=272, filesize=9.5 K 2023-04-25 09:56:00,697 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 6b4247a1e1c946a4543dfbc0edb52631 in 852ms, sequenceid=272, compaction requested=true 2023-04-25 09:56:00,697 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:00,697 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:00,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263ca84e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:00,703 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:00,703 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,704 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:00,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:00,706 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:00,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:00,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:00,708 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:00,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:00,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:00,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:00,708 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:00,709 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,709 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:00,709 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:00,710 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:00,713 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,718 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,718 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:00,718 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:00,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:00,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:00,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:00,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:00,719 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:00,719 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:00,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:00,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:00,819 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:00,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:00,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:00,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:00,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:00,822 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:00,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:00,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:00,822 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:00,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:00,824 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:00,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:00,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:00,824 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:00,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:00,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:00,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:00,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:00,825 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:00,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:00,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:00,826 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-25 09:56:00,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:00,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:00,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:00,826 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:00,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:00,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:00,834 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/b3f391b239974f6dbbc2b839bcfb6bcf 2023-04-25 09:56:01,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:01,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:01,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416621130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54364 deadline: 1682416621131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416621139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416621139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416621240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416621240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,244 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/7adc490af75e49e3b3f613772a6f0e3a 2023-04-25 09:56:01,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416621245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416621248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416621445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54396 deadline: 1682416621446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416621455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416621455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416621539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,546 DEBUG [Thread-3205] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:56:01,654 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/e806777aac174a1f8361b7ddc1238cbb 2023-04-25 09:56:01,658 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/b3f391b239974f6dbbc2b839bcfb6bcf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b3f391b239974f6dbbc2b839bcfb6bcf 2023-04-25 09:56:01,662 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b3f391b239974f6dbbc2b839bcfb6bcf, entries=150, sequenceid=309, filesize=11.9 K 2023-04-25 09:56:01,662 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/7adc490af75e49e3b3f613772a6f0e3a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7adc490af75e49e3b3f613772a6f0e3a 2023-04-25 09:56:01,665 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7adc490af75e49e3b3f613772a6f0e3a, entries=150, sequenceid=309, filesize=11.9 K 2023-04-25 09:56:01,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/e806777aac174a1f8361b7ddc1238cbb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e806777aac174a1f8361b7ddc1238cbb 2023-04-25 09:56:01,670 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e806777aac174a1f8361b7ddc1238cbb, entries=150, sequenceid=309, filesize=11.9 K 2023-04-25 09:56:01,671 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 6b4247a1e1c946a4543dfbc0edb52631 in 845ms, sequenceid=309, compaction requested=true 2023-04-25 09:56:01,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:01,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:01,673 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:01,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3edfc255[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:01,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:01,674 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:01,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:01,678 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:01,681 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:01,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:01,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:01,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:01,681 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:01,682 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,682 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:01,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,682 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:01,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:01,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:01,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,687 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:01,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:01,687 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:01,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:01,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:01,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:01,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:01,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:01,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:01,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:01,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:01,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/94db529db4b94da4acde962e9186fc81 2023-04-25 09:56:01,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/36f48327371d44f4b838ae6666641890 2023-04-25 09:56:01,787 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:01,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:01,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:01,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:01,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:01,792 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:01,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:01,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:01,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:01,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,794 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:01,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:01,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,794 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:01,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:01,800 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:01,800 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:01,800 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:01,800 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:01,800 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:01,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416561777, max timestamp=1682416561800Num uniques 859; 2023-04-25 09:56:01,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416561777, max timestamp=1682416561800Num uniques 859; 2023-04-25 09:56:01,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:56:01,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:56:01,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682416561777, max timestamp=1682416561800Num uniques 859; 2023-04-25 09:56:01,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:56:01,804 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,805 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,805 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6900d349[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:01,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:01,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,808 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:01,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:01,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:01,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:01,813 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:01,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:01,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:01,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:01,813 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:01,815 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,815 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,816 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:01,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,817 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:01,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:01,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:01,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:01,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:01,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:01,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,821 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:01,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:01,821 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:01,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:01,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416621860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416621861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416621878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416621879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,921 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:01,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:01,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:01,926 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:01,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:01,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:01,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:01,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:01,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:01,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:01,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:01,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:01,930 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:01,930 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:01,930 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:01,930 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:01,932 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6717730c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:01,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:01,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,937 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:01,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:01,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:01,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:01,942 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:01,942 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:01,942 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:01,942 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:01,942 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:01,944 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,944 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,944 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:01,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:01,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,946 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:01,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,950 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:01,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:01,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:01,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:01,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:01,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:01,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:01,951 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:01,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,951 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:01,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:01,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:01,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:01,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416621978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:01,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:01,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416621979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416621994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416621994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,051 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:02,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:02,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:02,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:02,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:02,054 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:02,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,057 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:02,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:02,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:02,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:02,059 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:02,061 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24aad978[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:02,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:02,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:02,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:02,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:02,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:02,070 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:02,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:02,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:02,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:02,070 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:02,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,074 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,078 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:02,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,078 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:02,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/abc7c4b75c004b59889d3189166bc875 2023-04-25 09:56:02,180 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:02,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:02,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/94db529db4b94da4acde962e9186fc81 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/94db529db4b94da4acde962e9186fc81 2023-04-25 09:56:02,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:02,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:02,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:02,184 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/94db529db4b94da4acde962e9186fc81, entries=150, sequenceid=319, filesize=11.9 K 2023-04-25 09:56:02,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,186 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/36f48327371d44f4b838ae6666641890 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36f48327371d44f4b838ae6666641890 2023-04-25 09:56:02,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:02,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:02,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:02,188 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,188 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:02,188 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:02,188 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:02,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36f48327371d44f4b838ae6666641890, entries=100, sequenceid=319, filesize=9.5 K 2023-04-25 09:56:02,189 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/abc7c4b75c004b59889d3189166bc875 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/abc7c4b75c004b59889d3189166bc875 2023-04-25 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64276ad2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:02,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:02,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/abc7c4b75c004b59889d3189166bc875, entries=100, sequenceid=319, filesize=9.5 K 2023-04-25 09:56:02,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 6b4247a1e1c946a4543dfbc0edb52631 in 440ms, sequenceid=319, compaction requested=true 2023-04-25 09:56:02,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:02,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:02,194 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:02,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,195 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:02,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:02,195 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:02,195 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:02,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:02,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:02,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:02,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,196 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,196 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68254 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:02,197 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,197 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,197 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/cdee0da979254bab955e5b55945651bd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/86581cf452854997ab3c35e97c6a97d0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6a7f07ef4fc04d34860bff301833cce8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/353fc21a742c4547bcbc88cbf1b936c6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7adc490af75e49e3b3f613772a6f0e3a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36f48327371d44f4b838ae6666641890] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=66.7 K 2023-04-25 09:56:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:02,197 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70694 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:02,197 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:02,197 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,198 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/53aecdeecf5c47689b05ac89c98c749f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b8e8b42ab2004cd7868f529c850d1b6c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b59368168cbd424a810c73fd245d5764, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7ec38f999f614856858340ed15484f72, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b3f391b239974f6dbbc2b839bcfb6bcf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/94db529db4b94da4acde962e9186fc81] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=69.0 K 2023-04-25 09:56:02,198 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting cdee0da979254bab955e5b55945651bd, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682416557405 2023-04-25 09:56:02,198 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 53aecdeecf5c47689b05ac89c98c749f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682416557405 2023-04-25 09:56:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,198 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 86581cf452854997ab3c35e97c6a97d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682416557734 2023-04-25 09:56:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,198 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b8e8b42ab2004cd7868f529c850d1b6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682416557734 2023-04-25 09:56:02,199 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6a7f07ef4fc04d34860bff301833cce8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682416558442 2023-04-25 09:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,199 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b59368168cbd424a810c73fd245d5764, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682416558442 2023-04-25 09:56:02,199 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 353fc21a742c4547bcbc88cbf1b936c6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682416559590 2023-04-25 09:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,200 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 7ec38f999f614856858340ed15484f72, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682416559590 2023-04-25 09:56:02,200 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7adc490af75e49e3b3f613772a6f0e3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682416559970 2023-04-25 09:56:02,201 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting b3f391b239974f6dbbc2b839bcfb6bcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682416559970 2023-04-25 09:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:02,201 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 36f48327371d44f4b838ae6666641890, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682416561137 2023-04-25 09:56:02,201 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:02,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:02,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:02,201 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 94db529db4b94da4acde962e9186fc81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682416561137 2023-04-25 09:56:02,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:02,201 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:02,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:02,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=154.31 KB heapSize=430.92 KB 2023-04-25 09:56:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:02,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:56:02,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:02,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:56:02,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:02,209 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:56:02,210 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,210 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,210 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,210 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,210 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:02,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:02,210 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,211 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:02,211 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:02,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:02,214 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#465 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:02,215 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#467 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:02,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416622214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416622221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,236 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/612b34724bc041f0b11a62555708e99e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/612b34724bc041f0b11a62555708e99e 2023-04-25 09:56:02,238 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/898569fe91274af4b79bdd6d7a1a9417 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/898569fe91274af4b79bdd6d7a1a9417 2023-04-25 09:56:02,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416622224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416622225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,242 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 898569fe91274af4b79bdd6d7a1a9417(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:02,242 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:02,242 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=10, startTime=1682416562194; duration=0sec 2023-04-25 09:56:02,242 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:02,242 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:02,243 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 612b34724bc041f0b11a62555708e99e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:02,243 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:02,243 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=10, startTime=1682416562194; duration=0sec 2023-04-25 09:56:02,243 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:02,243 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68254 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:02,243 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:02,244 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,244 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/5e44fb879bce4660bf0061e313a54f46, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d7a0e7aa72fc434ea74092c6bc297a56, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/984efee93dc84de2b6139c526710eec3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3530c8f9348241aeb8b920d85dfb2238, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e806777aac174a1f8361b7ddc1238cbb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/abc7c4b75c004b59889d3189166bc875] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=66.7 K 2023-04-25 09:56:02,244 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5e44fb879bce4660bf0061e313a54f46, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682416557405 2023-04-25 09:56:02,244 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d7a0e7aa72fc434ea74092c6bc297a56, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1682416557734 2023-04-25 09:56:02,244 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 984efee93dc84de2b6139c526710eec3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682416558442 2023-04-25 09:56:02,245 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3530c8f9348241aeb8b920d85dfb2238, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682416559590 2023-04-25 09:56:02,245 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e806777aac174a1f8361b7ddc1238cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1682416559970 2023-04-25 09:56:02,245 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting abc7c4b75c004b59889d3189166bc875, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682416561137 2023-04-25 09:56:02,251 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:02,311 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:02,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:02,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:02,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:02,315 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:02,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:02,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:02,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,317 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:02,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:02,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:02,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:02,321 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b527784[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:02,323 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:02,323 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:02,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:02,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:02,325 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,325 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:02,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:02,328 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:02,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:02,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:02,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:02,329 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:02,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416622326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,331 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,331 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,332 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:02,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:02,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:02,337 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:02,337 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:02,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54306 deadline: 1682416622337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416622340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416622342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,437 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:02,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:02,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:02,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:02,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:02,440 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:02,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:02,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:02,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,441 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:02,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:02,444 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,444 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:02,444 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:02,444 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:02,453 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,453 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,453 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,453 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5701eaf1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:02,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:02,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:02,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:02,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:02,460 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:02,462 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:02,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:02,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:02,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:02,462 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:02,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:02,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:02,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:02,470 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:02,470 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:02,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416622533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416622541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416622542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416622548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,570 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:02,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:02,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:02,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:02,574 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:02,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:02,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:02,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:02,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:02,580 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25928870[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:02,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:02,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:02,587 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:02,587 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:02,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:02,588 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:02,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:02,591 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:02,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:02,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:02,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:02,591 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:02,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:02,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:02,594 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,595 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:02,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:02,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:02,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:02,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:02,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:02,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:02,599 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:02,599 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:02,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/24450e65bf7b451c911aaabee61b8620 2023-04-25 09:56:02,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/0ce68494b7ac4e2a8fa855d08205539f 2023-04-25 09:56:02,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3ce90875c72f41349e68c0dbdea368f9 2023-04-25 09:56:02,666 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/e0085c70a11f4f9285d326b95426cf17 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e0085c70a11f4f9285d326b95426cf17 2023-04-25 09:56:02,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/24450e65bf7b451c911aaabee61b8620 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/24450e65bf7b451c911aaabee61b8620 2023-04-25 09:56:02,672 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into e0085c70a11f4f9285d326b95426cf17(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:02,672 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:02,672 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=10, startTime=1682416562195; duration=0sec 2023-04-25 09:56:02,672 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:02,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/24450e65bf7b451c911aaabee61b8620, entries=250, sequenceid=357, filesize=16.7 K 2023-04-25 09:56:02,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/0ce68494b7ac4e2a8fa855d08205539f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0ce68494b7ac4e2a8fa855d08205539f 2023-04-25 09:56:02,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0ce68494b7ac4e2a8fa855d08205539f, entries=150, sequenceid=357, filesize=11.9 K 2023-04-25 09:56:02,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3ce90875c72f41349e68c0dbdea368f9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3ce90875c72f41349e68c0dbdea368f9 2023-04-25 09:56:02,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3ce90875c72f41349e68c0dbdea368f9, entries=150, sequenceid=357, filesize=11.9 K 2023-04-25 09:56:02,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~451.27 KB/462096, currentSize=33.54 KB/34350 for 6b4247a1e1c946a4543dfbc0edb52631 in 477ms, sequenceid=357, compaction requested=false 2023-04-25 09:56:02,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:02,699 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:02,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:02,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:02,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:02,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:02,702 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:02,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:02,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:02,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,705 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:02,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:02,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:02,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:02,707 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:02,707 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:02,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:02,718 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/7b3383f2ce8c45fe80830a88442f812b 2023-04-25 09:56:02,735 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/60e129145e10419fa15e12dcb7bc3917 2023-04-25 09:56:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:02,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:02,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416562864, max timestamp=1682416562879Num uniques 859; 2023-04-25 09:56:02,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416562864, max timestamp=1682416562879Num uniques 859; 2023-04-25 09:56:02,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416562864, max timestamp=1682416562879Num uniques 859; 2023-04-25 09:56:02,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:02,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:02,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:02,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416622926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416622926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416622926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:02,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:02,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416622927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416623034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416623034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416623035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416623035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,144 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/617b93ddbafa44bc8095cc64ac4b0914 2023-04-25 09:56:03,148 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/7b3383f2ce8c45fe80830a88442f812b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7b3383f2ce8c45fe80830a88442f812b 2023-04-25 09:56:03,152 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7b3383f2ce8c45fe80830a88442f812b, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 09:56:03,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/60e129145e10419fa15e12dcb7bc3917 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/60e129145e10419fa15e12dcb7bc3917 2023-04-25 09:56:03,157 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/60e129145e10419fa15e12dcb7bc3917, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 09:56:03,158 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/617b93ddbafa44bc8095cc64ac4b0914 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/617b93ddbafa44bc8095cc64ac4b0914 2023-04-25 09:56:03,162 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/617b93ddbafa44bc8095cc64ac4b0914, entries=150, sequenceid=368, filesize=11.9 K 2023-04-25 09:56:03,162 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 6b4247a1e1c946a4543dfbc0edb52631 in 455ms, sequenceid=368, compaction requested=true 2023-04-25 09:56:03,162 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:03,162 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,163 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,165 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3c07ea[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,167 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,170 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,176 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,177 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,177 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:03,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-25 09:56:03,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:03,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:56:03,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:03,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:56:03,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:03,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-25 09:56:03,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416623250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416623250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416623251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416623259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,277 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:03,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:03,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:03,280 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:03,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:03,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:03,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,282 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:03,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:03,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:03,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:03,283 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,283 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:03,284 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:03,284 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,285 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@624aa17e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,290 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,291 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:03,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,294 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,294 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,299 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,301 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,301 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:03,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416623360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416623360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416623360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416623364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,402 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:03,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:03,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:03,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:03,405 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,408 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:03,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:03,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:03,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:03,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:03,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:03,410 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,410 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:03,410 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:03,410 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,411 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ac499a4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,413 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,413 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,415 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:03,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,418 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,418 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,422 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,424 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,424 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,424 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,424 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,425 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,425 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:03,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,426 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,525 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:03,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:03,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:03,529 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:03,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:03,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:03,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,529 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,530 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:03,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:03,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:03,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:03,532 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:03,532 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:03,533 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,533 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,534 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,536 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ba45784[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,537 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,538 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,540 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,542 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,543 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,543 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,543 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,544 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,544 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,544 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,545 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,549 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,549 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,549 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,549 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:03,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416623568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416623568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416623568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416623568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,650 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:03,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:03,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:03,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:03,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:03,653 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2aedf8654f7d4f12b36ecaa7e16c067c 2023-04-25 09:56:03,656 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,656 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:03,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:03,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:03,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:03,661 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:03,661 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:03,661 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,661 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:03,661 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:03,662 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,663 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11bd024d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,665 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,665 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,666 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,667 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,667 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,670 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,671 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,671 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:03,679 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,679 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,679 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,779 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:03,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:03,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:03,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:03,783 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:03,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:03,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:03,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,785 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:03,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:03,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:03,786 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:03,786 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,786 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:03,786 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:03,786 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:03,786 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,787 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:03,787 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:03,787 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,787 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,787 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,787 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,787 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,787 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,789 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,789 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26cb1055[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,798 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,798 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,798 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,798 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,798 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,799 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,799 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,802 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,802 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,802 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,802 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,802 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,804 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,804 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,804 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,808 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,808 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:03,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,809 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,809 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,810 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,810 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416623877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416623877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416623878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416623878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,910 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:03,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:03,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:03,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:03,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:03,913 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:03,913 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,913 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:03,913 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:03,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,914 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,918 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,918 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,918 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,918 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:03,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:03,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:03,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:03,920 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:03,921 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:03,921 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:03,922 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,925 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,925 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,925 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385ca491[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:03,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:03,926 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,928 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:03,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,928 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:03,929 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:03,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:03,932 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:03,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:03,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:03,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:03,932 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:03,934 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:03,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:03,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:03,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:03,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:03,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:03,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:03,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:03,939 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:03,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:03,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:03,940 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:03,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,940 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:03,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:03,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,040 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:04,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:04,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:04,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:04,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:04,043 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,046 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:04,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:04,047 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,047 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:04,047 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:04,047 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,048 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:04,048 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:04,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116ddc69[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:04,051 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:04,051 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:04,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:04,054 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:04,054 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:04,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:04,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:04,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:04,055 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:04,056 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,056 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,056 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,058 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,059 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:04,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:04,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:04,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,066 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:04,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,066 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:04,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/da4ffc59dffb43dea0672567271fb769 2023-04-25 09:56:04,167 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:04,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:04,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:04,170 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:04,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:04,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:04,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,172 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:04,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:04,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:04,174 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,174 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:04,174 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:04,174 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:04,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683b0b63[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:04,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:04,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,179 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,179 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:04,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:04,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:04,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:04,181 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,183 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,183 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,183 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:04,184 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:04,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:04,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:04,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:04,185 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:04,186 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,186 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,186 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,187 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:04,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,191 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:04,191 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:04,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:04,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,291 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:04,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:04,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:04,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:04,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:04,295 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:04,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:04,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:04,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,297 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:04,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:04,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:04,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:04,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:04,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:04,299 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:04,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,299 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:04,300 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:04,300 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,301 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160cf789[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:04,303 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:04,303 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,304 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:04,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:04,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:04,307 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:04,311 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:04,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:04,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:04,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:04,311 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:04,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,314 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,317 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,317 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,317 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:04,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:04,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:04,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,318 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:04,318 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:04,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416624388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416624388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416624389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:04,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416624391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,418 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:04,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:04,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:04,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:04,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:04,421 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:04,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:04,421 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:04,422 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,422 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,423 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,423 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,423 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,423 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:04,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:04,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:04,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:04,429 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:04,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b29a1f5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:04,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:04,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:04,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:04,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:04,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:04,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:04,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,433 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:04,435 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:04,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:04,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:04,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:04,436 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:04,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:04,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:04,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,438 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:04,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:04,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:04,446 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:04,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:04,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:04,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,448 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:04,448 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:04,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/fb832c3cc13d46bbb550e44a5cbcbd52 2023-04-25 09:56:04,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2aedf8654f7d4f12b36ecaa7e16c067c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2aedf8654f7d4f12b36ecaa7e16c067c 2023-04-25 09:56:04,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2aedf8654f7d4f12b36ecaa7e16c067c, entries=200, sequenceid=406, filesize=14.3 K 2023-04-25 09:56:04,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/da4ffc59dffb43dea0672567271fb769 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/da4ffc59dffb43dea0672567271fb769 2023-04-25 09:56:04,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/da4ffc59dffb43dea0672567271fb769, entries=150, sequenceid=406, filesize=11.9 K 2023-04-25 09:56:04,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/fb832c3cc13d46bbb550e44a5cbcbd52 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fb832c3cc13d46bbb550e44a5cbcbd52 2023-04-25 09:56:04,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fb832c3cc13d46bbb550e44a5cbcbd52, entries=150, sequenceid=406, filesize=11.9 K 2023-04-25 09:56:04,494 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for 6b4247a1e1c946a4543dfbc0edb52631 in 1247ms, sequenceid=406, compaction requested=true 2023-04-25 09:56:04,494 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:04,494 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:04,494 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:04,494 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:04,494 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:04,494 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:04,495 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56644 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:04,495 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:04,495 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,495 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/612b34724bc041f0b11a62555708e99e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/24450e65bf7b451c911aaabee61b8620, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7b3383f2ce8c45fe80830a88442f812b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2aedf8654f7d4f12b36ecaa7e16c067c] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=55.3 K 2023-04-25 09:56:04,496 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:04,496 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:04,496 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,496 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/898569fe91274af4b79bdd6d7a1a9417, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0ce68494b7ac4e2a8fa855d08205539f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/60e129145e10419fa15e12dcb7bc3917, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/da4ffc59dffb43dea0672567271fb769] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=48.2 K 2023-04-25 09:56:04,496 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 612b34724bc041f0b11a62555708e99e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682416559971 2023-04-25 09:56:04,497 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 898569fe91274af4b79bdd6d7a1a9417, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682416559971 2023-04-25 09:56:04,497 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 24450e65bf7b451c911aaabee61b8620, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682416561828 2023-04-25 09:56:04,497 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0ce68494b7ac4e2a8fa855d08205539f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682416561828 2023-04-25 09:56:04,498 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 7b3383f2ce8c45fe80830a88442f812b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682416562212 2023-04-25 09:56:04,498 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2aedf8654f7d4f12b36ecaa7e16c067c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682416562897 2023-04-25 09:56:04,499 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 60e129145e10419fa15e12dcb7bc3917, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682416562212 2023-04-25 09:56:04,500 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting da4ffc59dffb43dea0672567271fb769, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682416562897 2023-04-25 09:56:04,510 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#478 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:04,510 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#477 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:04,529 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/c0ee0fef583d4d899561eaad29daba65 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0ee0fef583d4d899561eaad29daba65 2023-04-25 09:56:04,536 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into c0ee0fef583d4d899561eaad29daba65(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:04,536 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:04,536 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=12, startTime=1682416564494; duration=0sec 2023-04-25 09:56:04,536 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:04,536 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:04,538 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:04,538 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:04,538 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,539 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e0085c70a11f4f9285d326b95426cf17, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3ce90875c72f41349e68c0dbdea368f9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/617b93ddbafa44bc8095cc64ac4b0914, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fb832c3cc13d46bbb550e44a5cbcbd52] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=48.2 K 2023-04-25 09:56:04,539 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e0085c70a11f4f9285d326b95426cf17, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682416559971 2023-04-25 09:56:04,539 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3ce90875c72f41349e68c0dbdea368f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1682416561828 2023-04-25 09:56:04,539 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 617b93ddbafa44bc8095cc64ac4b0914, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1682416562212 2023-04-25 09:56:04,540 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting fb832c3cc13d46bbb550e44a5cbcbd52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682416562897 2023-04-25 09:56:04,548 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/e8dc37c5b4c14ed4acb087251b3ee4c0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/e8dc37c5b4c14ed4acb087251b3ee4c0 2023-04-25 09:56:04,548 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:04,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:04,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:04,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:04,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:04,552 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:04,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:04,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:04,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,554 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:04,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:04,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:04,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:04,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:04,558 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#479 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:04,558 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into e8dc37c5b4c14ed4acb087251b3ee4c0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:04,558 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:04,558 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=12, startTime=1682416564494; duration=0sec 2023-04-25 09:56:04,558 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:04,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:04,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:04,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:04,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:04,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:04,564 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 09:56:04,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:04,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:04,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:04,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:04,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:04,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:04,582 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2a6f3e737aa14d0abe1e21e94aa5d2cc 2023-04-25 09:56:04,598 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/65613ed82ca3499688cf2a1164809adf 2023-04-25 09:56:04,972 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/23de689335f84c2a9bbf9476ba8289cd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/23de689335f84c2a9bbf9476ba8289cd 2023-04-25 09:56:04,976 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 23de689335f84c2a9bbf9476ba8289cd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:04,976 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:04,976 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=12, startTime=1682416564494; duration=0sec 2023-04-25 09:56:04,976 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:05,018 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/50238a0a0b5543db8346d71ca7692e04 2023-04-25 09:56:05,022 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2a6f3e737aa14d0abe1e21e94aa5d2cc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a6f3e737aa14d0abe1e21e94aa5d2cc 2023-04-25 09:56:05,025 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a6f3e737aa14d0abe1e21e94aa5d2cc, entries=100, sequenceid=415, filesize=9.5 K 2023-04-25 09:56:05,026 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/65613ed82ca3499688cf2a1164809adf as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/65613ed82ca3499688cf2a1164809adf 2023-04-25 09:56:05,029 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/65613ed82ca3499688cf2a1164809adf, entries=100, sequenceid=415, filesize=9.5 K 2023-04-25 09:56:05,030 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/50238a0a0b5543db8346d71ca7692e04 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/50238a0a0b5543db8346d71ca7692e04 2023-04-25 09:56:05,035 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/50238a0a0b5543db8346d71ca7692e04, entries=100, sequenceid=415, filesize=9.5 K 2023-04-25 09:56:05,035 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6b4247a1e1c946a4543dfbc0edb52631 in 471ms, sequenceid=415, compaction requested=false 2023-04-25 09:56:05,035 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,035 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,035 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,037 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3458b26d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,039 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,043 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,043 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,045 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,048 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,048 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,048 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,050 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,050 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,051 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,055 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,055 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,055 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,055 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,055 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,055 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,055 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,056 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:05,156 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,159 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,159 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,161 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,166 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@488d796[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,170 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,173 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,176 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,176 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,178 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,178 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,183 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:05,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,183 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,183 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,283 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,287 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,292 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,292 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,292 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,292 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,295 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7549d831[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,297 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,297 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,298 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,298 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,300 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,304 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,304 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,305 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,305 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,306 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,306 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,306 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,307 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,310 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,310 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,310 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,310 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:05,403 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:05,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:05,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:05,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:05,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:05,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:05,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:05,410 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/1472cdfe0b2a4e37a56e899914d15ff8 2023-04-25 09:56:05,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,415 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,417 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,417 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,418 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,419 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,419 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,419 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,420 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:05,420 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/ca90b95d6daf4ba8960870a9bad7c0f7 2023-04-25 09:56:05,425 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7369f964[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,427 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,427 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,429 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,431 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/b7d0a70a22fa40bdad217dd7c1d0c53f 2023-04-25 09:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,435 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,435 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,435 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682416565421, max timestamp=1682416565437Num uniques 859; 2023-04-25 09:56:05,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:56:05,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682416565421, max timestamp=1682416565437Num uniques 859; 2023-04-25 09:56:05,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:56:05,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1682416565421, max timestamp=1682416565437Num uniques 859; 2023-04-25 09:56:05,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-25 09:56:05,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/1472cdfe0b2a4e37a56e899914d15ff8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/1472cdfe0b2a4e37a56e899914d15ff8 2023-04-25 09:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,442 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/1472cdfe0b2a4e37a56e899914d15ff8, entries=250, sequenceid=428, filesize=16.7 K 2023-04-25 09:56:05,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/ca90b95d6daf4ba8960870a9bad7c0f7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ca90b95d6daf4ba8960870a9bad7c0f7 2023-04-25 09:56:05,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,445 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,446 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,446 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,446 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,446 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:05,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ca90b95d6daf4ba8960870a9bad7c0f7, entries=150, sequenceid=428, filesize=11.9 K 2023-04-25 09:56:05,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/b7d0a70a22fa40bdad217dd7c1d0c53f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b7d0a70a22fa40bdad217dd7c1d0c53f 2023-04-25 09:56:05,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b7d0a70a22fa40bdad217dd7c1d0c53f, entries=150, sequenceid=428, filesize=11.9 K 2023-04-25 09:56:05,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=73.80 KB/75570 for 6b4247a1e1c946a4543dfbc0edb52631 in 51ms, sequenceid=428, compaction requested=true 2023-04-25 09:56:05,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:05,454 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:05,454 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:05,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:56:05,455 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39710 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:05,455 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:05,455 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:05,455 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:05,455 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,455 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,455 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0ee0fef583d4d899561eaad29daba65, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a6f3e737aa14d0abe1e21e94aa5d2cc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/1472cdfe0b2a4e37a56e899914d15ff8] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=38.8 K 2023-04-25 09:56:05,455 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/e8dc37c5b4c14ed4acb087251b3ee4c0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/65613ed82ca3499688cf2a1164809adf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ca90b95d6daf4ba8960870a9bad7c0f7] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=34.0 K 2023-04-25 09:56:05,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:05,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=80.51 KB heapSize=211.55 KB 2023-04-25 09:56:05,456 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting c0ee0fef583d4d899561eaad29daba65, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682416562897 2023-04-25 09:56:05,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:05,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:05,456 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e8dc37c5b4c14ed4acb087251b3ee4c0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682416562897 2023-04-25 09:56:05,456 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2a6f3e737aa14d0abe1e21e94aa5d2cc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1682416563248 2023-04-25 09:56:05,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:05,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:05,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:05,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:05,456 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 65613ed82ca3499688cf2a1164809adf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1682416563248 2023-04-25 09:56:05,457 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1472cdfe0b2a4e37a56e899914d15ff8, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682416565395 2023-04-25 09:56:05,457 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ca90b95d6daf4ba8960870a9bad7c0f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682416565395 2023-04-25 09:56:05,469 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#487 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:05,470 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:05,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/cfd9a40a9e904afdac1b07c281e380da 2023-04-25 09:56:05,486 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/c0c9539fa76947e982c57bead439b4ad as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0c9539fa76947e982c57bead439b4ad 2023-04-25 09:56:05,490 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into c0c9539fa76947e982c57bead439b4ad(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:05,490 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,490 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=13, startTime=1682416565454; duration=0sec 2023-04-25 09:56:05,490 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:05,490 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:05,492 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:05,492 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:05,492 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,492 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/23de689335f84c2a9bbf9476ba8289cd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/50238a0a0b5543db8346d71ca7692e04, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b7d0a70a22fa40bdad217dd7c1d0c53f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=34.0 K 2023-04-25 09:56:05,496 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 23de689335f84c2a9bbf9476ba8289cd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1682416562897 2023-04-25 09:56:05,497 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 50238a0a0b5543db8346d71ca7692e04, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1682416563248 2023-04-25 09:56:05,497 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b7d0a70a22fa40bdad217dd7c1d0c53f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682416565395 2023-04-25 09:56:05,500 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/655c8fd156d443748c8c70b0d50e0866 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/655c8fd156d443748c8c70b0d50e0866 2023-04-25 09:56:05,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416625491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416625493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416625494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,504 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 655c8fd156d443748c8c70b0d50e0866(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:05,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,504 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416625495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,504 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=13, startTime=1682416565454; duration=0sec 2023-04-25 09:56:05,504 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:05,512 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:05,546 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,550 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,553 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:05,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,556 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3e6be[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,560 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,560 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,565 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,565 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,566 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,568 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,571 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,571 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,571 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,571 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:05,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,572 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,572 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416625604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416625605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416625605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416625605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,672 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,675 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,676 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,677 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,677 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,678 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,678 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,679 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,679 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:05,679 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a5b515[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,689 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,694 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,694 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,694 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,698 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:05,702 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,702 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,802 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,806 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,808 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,808 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,808 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,808 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,810 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,810 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,810 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:05,810 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:05,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:05,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b166560[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:05,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:05,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416625815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416625815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416625815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:05,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416625815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:05,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:05,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:05,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,822 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:05,827 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:05,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:05,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:05,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:05,827 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:05,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:05,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,833 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:05,835 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:05,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:05,835 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:05,836 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:05,836 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:05,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:05,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/9f7c6c58640345c8ac9e71b0cd102ad4 2023-04-25 09:56:05,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/07bd034e0a4448dba97f852451219300 2023-04-25 09:56:05,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/cfd9a40a9e904afdac1b07c281e380da as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/cfd9a40a9e904afdac1b07c281e380da 2023-04-25 09:56:05,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/cfd9a40a9e904afdac1b07c281e380da, entries=200, sequenceid=454, filesize=14.3 K 2023-04-25 09:56:05,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/9f7c6c58640345c8ac9e71b0cd102ad4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9f7c6c58640345c8ac9e71b0cd102ad4 2023-04-25 09:56:05,929 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/892c23eb428c4a9da11393fabf8b4b47 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/892c23eb428c4a9da11393fabf8b4b47 2023-04-25 09:56:05,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9f7c6c58640345c8ac9e71b0cd102ad4, entries=150, sequenceid=454, filesize=11.9 K 2023-04-25 09:56:05,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/07bd034e0a4448dba97f852451219300 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/07bd034e0a4448dba97f852451219300 2023-04-25 09:56:05,936 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:05,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:05,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/07bd034e0a4448dba97f852451219300, entries=150, sequenceid=454, filesize=11.9 K 2023-04-25 09:56:05,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~211.50 KB/216576, currentSize=107.34 KB/109920 for 6b4247a1e1c946a4543dfbc0edb52631 in 482ms, sequenceid=454, compaction requested=false 2023-04-25 09:56:05,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:05,940 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 892c23eb428c4a9da11393fabf8b4b47(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:05,940 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:05,940 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=13, startTime=1682416565454; duration=0sec 2023-04-25 09:56:05,940 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:05,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:05,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:05,941 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:05,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:05,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:05,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,943 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:05,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:05,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:05,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:05,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:05,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:05,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:05,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:05,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:05,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:05,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:05,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:05,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:05,956 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:05,956 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:05,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:05,956 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:56:05,956 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:05,956 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:05,957 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:05,957 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:05,957 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:05,957 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:05,980 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/ea6e5e458e9a4b87b2f7b509972832ef 2023-04-25 09:56:06,049 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/d1dca1c477d04c6aba744376664dd4bc 2023-04-25 09:56:06,100 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/22965ddafe2d44e482ace90c437e0811 2023-04-25 09:56:06,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/ea6e5e458e9a4b87b2f7b509972832ef as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ea6e5e458e9a4b87b2f7b509972832ef 2023-04-25 09:56:06,109 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ea6e5e458e9a4b87b2f7b509972832ef, entries=150, sequenceid=476, filesize=11.9 K 2023-04-25 09:56:06,110 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/d1dca1c477d04c6aba744376664dd4bc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d1dca1c477d04c6aba744376664dd4bc 2023-04-25 09:56:06,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:06,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:06,126 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d1dca1c477d04c6aba744376664dd4bc, entries=150, sequenceid=476, filesize=11.9 K 2023-04-25 09:56:06,127 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/22965ddafe2d44e482ace90c437e0811 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/22965ddafe2d44e482ace90c437e0811 2023-04-25 09:56:06,166 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/22965ddafe2d44e482ace90c437e0811, entries=150, sequenceid=476, filesize=11.9 K 2023-04-25 09:56:06,166 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=46.96 KB/48090 for 6b4247a1e1c946a4543dfbc0edb52631 in 210ms, sequenceid=476, compaction requested=true 2023-04-25 09:56:06,167 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:06,167 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:06,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:06,167 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:56:06,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:06,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:06,167 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:06,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:06,169 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,170 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:06,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@478811bf[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:06,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:06,172 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,173 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:06,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:06,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:06,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,173 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:06,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:06,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:06,175 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:06,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:06,177 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:06,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:06,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:06,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:06,177 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:06,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,180 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,182 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:06,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,184 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,185 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,185 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:06,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416626222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416566210, max timestamp=1682416566223Num uniques 859; 2023-04-25 09:56:06,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416566210, max timestamp=1682416566223Num uniques 859; 2023-04-25 09:56:06,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:56:06,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416566210, max timestamp=1682416566223Num uniques 859; 2023-04-25 09:56:06,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:56:06,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:56:06,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416626258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416626262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416626263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,285 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:06,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:06,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:06,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:06,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:06,288 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:06,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:06,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:06,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,290 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:06,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:06,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:06,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:06,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:06,291 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,291 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:06,291 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:06,291 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,291 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:06,292 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:06,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,293 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,293 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:06,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:06,296 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af3e9e6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:06,300 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:06,301 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:06,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,301 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,302 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,302 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:06,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:06,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:06,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:06,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:06,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:06,306 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:06,306 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:06,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,308 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,310 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,313 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,313 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,313 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,313 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:06,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:06,313 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,314 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,314 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,314 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,314 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,314 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,314 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:06,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:06,314 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:06,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416626326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416626366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416626371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416626372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,414 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:06,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:06,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:06,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:06,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:06,420 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:06,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:06,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:06,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,420 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,422 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,422 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,424 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:06,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:06,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:06,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:06,425 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:06,425 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:06,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:06,426 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:06,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:06,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:06,426 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:06,427 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,427 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,428 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:06,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3793e2b0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:06,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:06,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,432 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:06,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:06,433 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,433 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:06,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:06,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,437 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:06,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:06,440 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:06,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:06,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:06,440 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:06,440 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:06,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,442 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,444 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:06,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,447 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,447 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:06,447 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416626538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,547 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:06,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:06,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:06,550 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:06,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:06,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:06,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,551 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:06,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:06,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:06,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:06,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:06,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:06,554 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:06,554 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:06,556 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:06,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f66ac47[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:06,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:06,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:06,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,559 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:06,561 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:06,561 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,561 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,561 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:06,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:06,563 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:06,563 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:06,563 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:06,563 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:06,563 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:06,564 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:06,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,565 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,566 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:06,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,569 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:06,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,572 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:06,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:06,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:06,572 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:06,572 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:06,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:06,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416626575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416626578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416626581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/415ff5ce446b4036beb1ba8a4d203f5a 2023-04-25 09:56:06,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/7d6db2dd62284d77a798a39c0b3eb801 2023-04-25 09:56:06,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/552d9ef420ae485abc12edd1bcc098dd 2023-04-25 09:56:06,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/415ff5ce446b4036beb1ba8a4d203f5a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/415ff5ce446b4036beb1ba8a4d203f5a 2023-04-25 09:56:06,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/415ff5ce446b4036beb1ba8a4d203f5a, entries=350, sequenceid=487, filesize=21.4 K 2023-04-25 09:56:06,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/7d6db2dd62284d77a798a39c0b3eb801 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7d6db2dd62284d77a798a39c0b3eb801 2023-04-25 09:56:06,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7d6db2dd62284d77a798a39c0b3eb801, entries=150, sequenceid=487, filesize=11.9 K 2023-04-25 09:56:06,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/552d9ef420ae485abc12edd1bcc098dd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/552d9ef420ae485abc12edd1bcc098dd 2023-04-25 09:56:06,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/552d9ef420ae485abc12edd1bcc098dd, entries=150, sequenceid=487, filesize=11.9 K 2023-04-25 09:56:06,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 6b4247a1e1c946a4543dfbc0edb52631 in 484ms, sequenceid=487, compaction requested=true 2023-04-25 09:56:06,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:06,651 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:06,651 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:06,651 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:06,651 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:06,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:06,653 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:06,653 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:06,653 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,653 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/655c8fd156d443748c8c70b0d50e0866, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9f7c6c58640345c8ac9e71b0cd102ad4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d1dca1c477d04c6aba744376664dd4bc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7d6db2dd62284d77a798a39c0b3eb801] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=48.4 K 2023-04-25 09:56:06,653 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 61766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:06,653 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:06,654 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,654 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0c9539fa76947e982c57bead439b4ad, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/cfd9a40a9e904afdac1b07c281e380da, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ea6e5e458e9a4b87b2f7b509972832ef, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/415ff5ce446b4036beb1ba8a4d203f5a] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=60.3 K 2023-04-25 09:56:06,654 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 655c8fd156d443748c8c70b0d50e0866, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682416565395 2023-04-25 09:56:06,654 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9f7c6c58640345c8ac9e71b0cd102ad4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682416565448 2023-04-25 09:56:06,654 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c0c9539fa76947e982c57bead439b4ad, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682416565395 2023-04-25 09:56:06,654 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d1dca1c477d04c6aba744376664dd4bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682416565469 2023-04-25 09:56:06,655 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting cfd9a40a9e904afdac1b07c281e380da, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682416565437 2023-04-25 09:56:06,655 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7d6db2dd62284d77a798a39c0b3eb801, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416566128 2023-04-25 09:56:06,655 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ea6e5e458e9a4b87b2f7b509972832ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682416565469 2023-04-25 09:56:06,657 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 415ff5ce446b4036beb1ba8a4d203f5a, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416566125 2023-04-25 09:56:06,670 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#498 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:06,673 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#499 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:06,673 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:06,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:06,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:06,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:06,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:06,688 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:06,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:06,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:06,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:06,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:06,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:06,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:06,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:06,710 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/29268f33f31f41f0be35491c72bd5c31 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/29268f33f31f41f0be35491c72bd5c31 2023-04-25 09:56:06,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:06,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:06,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:06,712 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-25 09:56:06,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:06,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:06,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:06,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:06,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:06,713 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:06,715 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 29268f33f31f41f0be35491c72bd5c31(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:06,715 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:06,715 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=12, startTime=1682416566651; duration=0sec 2023-04-25 09:56:06,716 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:06,716 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:06,717 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:06,717 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:06,717 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:06,717 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/892c23eb428c4a9da11393fabf8b4b47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/07bd034e0a4448dba97f852451219300, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/22965ddafe2d44e482ace90c437e0811, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/552d9ef420ae485abc12edd1bcc098dd] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=48.4 K 2023-04-25 09:56:06,718 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 892c23eb428c4a9da11393fabf8b4b47, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1682416565395 2023-04-25 09:56:06,718 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 07bd034e0a4448dba97f852451219300, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1682416565448 2023-04-25 09:56:06,718 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 22965ddafe2d44e482ace90c437e0811, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682416565469 2023-04-25 09:56:06,719 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 552d9ef420ae485abc12edd1bcc098dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416566128 2023-04-25 09:56:06,725 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#501 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:06,745 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/63298f890e584945a3e4b9f07b627baa as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/63298f890e584945a3e4b9f07b627baa 2023-04-25 09:56:06,750 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 63298f890e584945a3e4b9f07b627baa(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:06,750 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:06,751 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=12, startTime=1682416566652; duration=0sec 2023-04-25 09:56:06,751 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:06,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:06,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:06,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416626886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416626887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416626891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416626892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:06,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:06,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416626993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416626999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416626999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,102 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/f0ebd472b00d46aea79e7496d60ca6b9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f0ebd472b00d46aea79e7496d60ca6b9 2023-04-25 09:56:07,106 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into f0ebd472b00d46aea79e7496d60ca6b9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:07,106 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:07,106 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=12, startTime=1682416566651; duration=0sec 2023-04-25 09:56:07,106 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:07,137 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/c0322923de2246e193b2840527279dc0 2023-04-25 09:56:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416627198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416627204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416627205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416627393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416627511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416627511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:07,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416627511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,548 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/2daa6869b77848bcb624941edae47a37 2023-04-25 09:56:07,560 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/599ee509772c4e2fa7bdad46afc6371f 2023-04-25 09:56:07,563 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/c0322923de2246e193b2840527279dc0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0322923de2246e193b2840527279dc0 2023-04-25 09:56:07,567 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0322923de2246e193b2840527279dc0, entries=150, sequenceid=520, filesize=11.9 K 2023-04-25 09:56:07,569 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/2daa6869b77848bcb624941edae47a37 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/2daa6869b77848bcb624941edae47a37 2023-04-25 09:56:07,573 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/2daa6869b77848bcb624941edae47a37, entries=150, sequenceid=520, filesize=11.9 K 2023-04-25 09:56:07,574 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/599ee509772c4e2fa7bdad46afc6371f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/599ee509772c4e2fa7bdad46afc6371f 2023-04-25 09:56:07,583 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/599ee509772c4e2fa7bdad46afc6371f, entries=150, sequenceid=520, filesize=11.9 K 2023-04-25 09:56:07,584 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 6b4247a1e1c946a4543dfbc0edb52631 in 872ms, sequenceid=520, compaction requested=false 2023-04-25 09:56:07,584 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:07,584 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:07,584 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:07,584 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:07,584 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,584 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:07,584 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:07,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:07,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,586 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:07,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:07,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:07,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78644ffd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:07,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:07,588 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,589 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:07,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,589 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:07,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:07,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:07,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:07,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:07,591 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:07,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:07,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:07,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:07,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:07,595 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:07,595 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:07,595 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:07,595 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:07,595 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:07,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:07,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:07,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:07,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:07,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:07,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:07,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:07,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:07,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:07,602 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:07,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:07,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:07,602 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:07,703 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:07,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:07,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:07,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:07,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:07,706 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:07,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:07,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:07,706 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:07,707 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,707 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:07,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:07,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:07,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:07,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:07,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:07,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:07,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:07,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:07,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:07,710 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:07,710 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:07,711 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:56:07,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:07,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:07,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:07,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:07,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:07,711 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:08,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:08,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:08,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416568045, max timestamp=1682416568053Num uniques 859; 2023-04-25 09:56:08,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:08,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416568045, max timestamp=1682416568053Num uniques 859; 2023-04-25 09:56:08,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416568045, max timestamp=1682416568053Num uniques 859; 2023-04-25 09:56:08,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:08,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:08,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416628080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416628085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416628085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,122 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/87e697611cb6489fb4f1c1bd8b253762 2023-04-25 09:56:08,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416628188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416628190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416628193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416628393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416628395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416628398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416628398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,535 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/46ffd14609024ced86b825ea234300bb 2023-04-25 09:56:08,548 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3bd25b11525341dd9f07025ba9696416 2023-04-25 09:56:08,551 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/87e697611cb6489fb4f1c1bd8b253762 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/87e697611cb6489fb4f1c1bd8b253762 2023-04-25 09:56:08,555 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/87e697611cb6489fb4f1c1bd8b253762, entries=150, sequenceid=534, filesize=11.9 K 2023-04-25 09:56:08,556 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/46ffd14609024ced86b825ea234300bb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/46ffd14609024ced86b825ea234300bb 2023-04-25 09:56:08,560 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/46ffd14609024ced86b825ea234300bb, entries=150, sequenceid=534, filesize=11.9 K 2023-04-25 09:56:08,561 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3bd25b11525341dd9f07025ba9696416 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3bd25b11525341dd9f07025ba9696416 2023-04-25 09:56:08,564 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3bd25b11525341dd9f07025ba9696416, entries=150, sequenceid=534, filesize=11.9 K 2023-04-25 09:56:08,564 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6b4247a1e1c946a4543dfbc0edb52631 in 853ms, sequenceid=534, compaction requested=true 2023-04-25 09:56:08,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:08,565 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:08,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:08,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:08,568 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,568 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:08,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:08,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d9d25ce[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:08,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:08,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,570 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:08,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:08,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:08,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:08,572 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:08,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:08,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:08,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:08,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:08,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:08,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:08,574 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:08,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:08,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:08,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:08,574 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:08,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:08,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:08,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:08,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:08,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:08,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:08,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:08,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:08,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:08,585 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,585 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:08,585 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:08,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:08,685 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:08,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:08,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:08,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:08,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:08,689 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:08,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:08,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:08,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:08,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:08,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,690 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:08,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:08,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:08,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:08,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:08,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:08,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:08,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:08,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:08,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:08,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:08,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:08,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:08,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:08,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:08,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:08,692 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 09:56:08,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:08,692 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:08,693 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:08,693 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:08,693 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:08,693 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:08,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:08,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:08,700 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/3368acc508864d12b4b27a841366fd2e 2023-04-25 09:56:08,711 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/852088a3a96b47e98fba8bae1d6f94b1 2023-04-25 09:56:08,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416628718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416628722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416628722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416628823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416628825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:08,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:08,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416628831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416629027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416629031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416629034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,122 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3d8c8b2f7529452c9476ba6f05f87309 2023-04-25 09:56:09,125 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/3368acc508864d12b4b27a841366fd2e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/3368acc508864d12b4b27a841366fd2e 2023-04-25 09:56:09,128 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/3368acc508864d12b4b27a841366fd2e, entries=150, sequenceid=568, filesize=11.9 K 2023-04-25 09:56:09,129 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/852088a3a96b47e98fba8bae1d6f94b1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/852088a3a96b47e98fba8bae1d6f94b1 2023-04-25 09:56:09,132 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/852088a3a96b47e98fba8bae1d6f94b1, entries=150, sequenceid=568, filesize=11.9 K 2023-04-25 09:56:09,133 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/3d8c8b2f7529452c9476ba6f05f87309 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3d8c8b2f7529452c9476ba6f05f87309 2023-04-25 09:56:09,136 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3d8c8b2f7529452c9476ba6f05f87309, entries=150, sequenceid=568, filesize=11.9 K 2023-04-25 09:56:09,136 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 6b4247a1e1c946a4543dfbc0edb52631 in 444ms, sequenceid=568, compaction requested=true 2023-04-25 09:56:09,136 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:09,136 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:09,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:09,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:09,137 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:09,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,138 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:09,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1533c6af[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:09,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:09,140 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:09,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:09,141 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:09,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:09,144 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:09,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,144 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:09,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:09,146 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:09,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:09,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:09,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:09,146 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:09,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,150 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:09,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:09,152 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,152 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,153 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:09,153 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:09,253 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:09,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:09,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:09,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:09,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:09,256 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:09,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:09,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:09,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:09,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,256 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,257 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,257 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,258 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,258 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:09,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:09,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:09,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:09,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:09,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:09,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:09,259 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:09,259 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,259 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:09,259 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:56:09,259 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:09,259 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:09,275 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/f67490c9374d4e959daf035139f78c60 2023-04-25 09:56:09,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:09,336 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:09,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416569355, max timestamp=1682416569390Num uniques 859; 2023-04-25 09:56:09,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416569355, max timestamp=1682416569390Num uniques 859; 2023-04-25 09:56:09,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:56:09,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:56:09,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416569355, max timestamp=1682416569390Num uniques 859; 2023-04-25 09:56:09,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:56:09,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416629434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416629435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416629442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416629545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416629545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416629548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,694 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/ccc0b01e8362402394cc93bbfd35e861 2023-04-25 09:56:09,704 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/ea71625a36a64f39822b2e80ce27b17b 2023-04-25 09:56:09,707 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/f67490c9374d4e959daf035139f78c60 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f67490c9374d4e959daf035139f78c60 2023-04-25 09:56:09,718 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f67490c9374d4e959daf035139f78c60, entries=150, sequenceid=579, filesize=11.9 K 2023-04-25 09:56:09,718 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/ccc0b01e8362402394cc93bbfd35e861 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ccc0b01e8362402394cc93bbfd35e861 2023-04-25 09:56:09,722 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ccc0b01e8362402394cc93bbfd35e861, entries=150, sequenceid=579, filesize=11.9 K 2023-04-25 09:56:09,722 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/ea71625a36a64f39822b2e80ce27b17b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/ea71625a36a64f39822b2e80ce27b17b 2023-04-25 09:56:09,726 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/ea71625a36a64f39822b2e80ce27b17b, entries=150, sequenceid=579, filesize=11.9 K 2023-04-25 09:56:09,726 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6b4247a1e1c946a4543dfbc0edb52631 in 467ms, sequenceid=579, compaction requested=true 2023-04-25 09:56:09,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:09,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:09,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:09,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:09,727 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:09,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:09,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7d54da[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:09,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:09,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,731 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:09,731 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,731 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:09,732 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:09,734 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:09,734 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:09,734 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:09,735 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:09,735 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:09,736 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,736 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:09,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:09,740 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,740 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,741 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,741 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:09,741 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:09,741 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:09,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-25 09:56:09,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:09,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:09,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:09,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:09,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:09,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:09,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416629779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416629786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416629787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,841 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:09,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:09,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:09,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:09,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:09,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:09,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:09,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:09,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:09,845 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,845 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,846 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:09,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:09,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:09,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:09,847 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:09,848 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,848 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:09,848 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:09,848 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:09,848 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:09,849 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,849 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ce36f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:09,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:09,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,854 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:09,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:09,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:09,857 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:09,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:09,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:09,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:09,857 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:09,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,861 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,868 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:09,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:09,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,868 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,868 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,868 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416629888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416629891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:09,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416629896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,968 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:09,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:09,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:09,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:09,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:09,971 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:09,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:09,972 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:09,972 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:09,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,973 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,974 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:09,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:09,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:09,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:09,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:09,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:09,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:09,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:09,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:09,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:09,978 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,979 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:09,979 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:09,979 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:09,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:09,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:09,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:09,979 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:09,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,980 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,981 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,981 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:09,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515b0553[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:09,983 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:09,983 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,984 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:09,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:09,984 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,984 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:09,984 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:09,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:09,989 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:09,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:09,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:09,989 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:09,989 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:09,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:09,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,993 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:09,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:09,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:09,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:09,995 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:09,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:09,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:09,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:09,995 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:09,996 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,096 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,099 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,101 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416630096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:10,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,103 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:10,104 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,106 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad6a769[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,108 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,108 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,109 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416630103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,111 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,111 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,111 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416630105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,113 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,118 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,118 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,118 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,118 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,118 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,119 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,119 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,119 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/62ed0f6cbf244084aeaca67de1244c95 2023-04-25 09:56:10,219 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,223 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,225 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:10,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,226 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,226 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,226 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:10,226 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb8f36d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,232 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,233 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,233 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,236 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,236 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:10,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,238 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,238 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,238 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,240 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,242 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,243 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,245 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,246 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,245 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,346 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,349 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,354 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,356 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,357 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:10,357 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,357 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,357 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,357 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,357 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,357 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d7d526e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,360 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,360 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,361 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,363 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,363 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,363 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,363 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,365 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,366 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,366 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,368 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,368 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,370 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,373 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:10,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,374 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,374 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416630405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416630411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416630414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:10,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54306 deadline: 1682416630418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,426 DEBUG [Thread-3207] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., hostname=jenkins-hbase4.apache.org,39407,1682416417123, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-25 09:56:10,474 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,477 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,479 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,481 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,481 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,481 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:10,481 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@405bf0f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,492 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,492 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,492 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:10,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,496 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,496 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,500 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,502 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,502 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,503 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,503 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,503 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/475f234febc54baf9b5042a391b8a1f5 2023-04-25 09:56:10,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/d2cd3db7f8114993a37c591a0c3148fc 2023-04-25 09:56:10,603 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/62ed0f6cbf244084aeaca67de1244c95 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/62ed0f6cbf244084aeaca67de1244c95 2023-04-25 09:56:10,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,606 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,607 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,607 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/62ed0f6cbf244084aeaca67de1244c95, entries=250, sequenceid=616, filesize=16.7 K 2023-04-25 09:56:10,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/475f234febc54baf9b5042a391b8a1f5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/475f234febc54baf9b5042a391b8a1f5 2023-04-25 09:56:10,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,610 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,610 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:10,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,611 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,611 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,611 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:10,611 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,611 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/475f234febc54baf9b5042a391b8a1f5, entries=150, sequenceid=616, filesize=11.9 K 2023-04-25 09:56:10,613 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/d2cd3db7f8114993a37c591a0c3148fc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d2cd3db7f8114993a37c591a0c3148fc 2023-04-25 09:56:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b07bd09[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,617 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,617 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d2cd3db7f8114993a37c591a0c3148fc, entries=150, sequenceid=616, filesize=11.9 K 2023-04-25 09:56:10,617 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,617 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 6b4247a1e1c946a4543dfbc0edb52631 in 856ms, sequenceid=616, compaction requested=true 2023-04-25 09:56:10,618 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:10,618 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:10,618 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:10,618 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:10,618 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:10,618 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:10,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,620 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78950 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:10,620 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:10,620 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,620 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74070 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:10,620 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f0ebd472b00d46aea79e7496d60ca6b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0322923de2246e193b2840527279dc0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/87e697611cb6489fb4f1c1bd8b253762, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/3368acc508864d12b4b27a841366fd2e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f67490c9374d4e959daf035139f78c60, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/62ed0f6cbf244084aeaca67de1244c95] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=77.1 K 2023-04-25 09:56:10,620 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:10,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,620 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,620 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/29268f33f31f41f0be35491c72bd5c31, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/2daa6869b77848bcb624941edae47a37, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/46ffd14609024ced86b825ea234300bb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/852088a3a96b47e98fba8bae1d6f94b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ccc0b01e8362402394cc93bbfd35e861, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/475f234febc54baf9b5042a391b8a1f5] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=72.3 K 2023-04-25 09:56:10,621 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f0ebd472b00d46aea79e7496d60ca6b9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416566128 2023-04-25 09:56:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,621 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 29268f33f31f41f0be35491c72bd5c31, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416566128 2023-04-25 09:56:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,621 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2daa6869b77848bcb624941edae47a37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682416566249 2023-04-25 09:56:10,621 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c0322923de2246e193b2840527279dc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682416566249 2023-04-25 09:56:10,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,622 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 46ffd14609024ced86b825ea234300bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682416566876 2023-04-25 09:56:10,622 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 87e697611cb6489fb4f1c1bd8b253762, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682416566876 2023-04-25 09:56:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:10,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,622 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,622 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 852088a3a96b47e98fba8bae1d6f94b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682416568059 2023-04-25 09:56:10,622 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,622 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,622 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,622 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,622 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3368acc508864d12b4b27a841366fd2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682416568059 2023-04-25 09:56:10,623 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ccc0b01e8362402394cc93bbfd35e861, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1682416568716 2023-04-25 09:56:10,623 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f67490c9374d4e959daf035139f78c60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1682416568716 2023-04-25 09:56:10,623 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 475f234febc54baf9b5042a391b8a1f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1682416569435 2023-04-25 09:56:10,624 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 62ed0f6cbf244084aeaca67de1244c95, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1682416569418 2023-04-25 09:56:10,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,626 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,627 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,629 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,632 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,632 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,633 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,633 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,637 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#516 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:10,638 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:10,657 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/9318f3466f9748a580e6d74cf491d9bc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/9318f3466f9748a580e6d74cf491d9bc 2023-04-25 09:56:10,661 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 9318f3466f9748a580e6d74cf491d9bc(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:10,661 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:10,661 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=10, startTime=1682416570618; duration=0sec 2023-04-25 09:56:10,661 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:10,661 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:10,664 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74070 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:10,664 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:10,664 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,664 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/63298f890e584945a3e4b9f07b627baa, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/599ee509772c4e2fa7bdad46afc6371f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3bd25b11525341dd9f07025ba9696416, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3d8c8b2f7529452c9476ba6f05f87309, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/ea71625a36a64f39822b2e80ce27b17b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d2cd3db7f8114993a37c591a0c3148fc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=72.3 K 2023-04-25 09:56:10,664 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 63298f890e584945a3e4b9f07b627baa, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682416566128 2023-04-25 09:56:10,665 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 599ee509772c4e2fa7bdad46afc6371f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1682416566249 2023-04-25 09:56:10,665 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3bd25b11525341dd9f07025ba9696416, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682416566876 2023-04-25 09:56:10,665 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3d8c8b2f7529452c9476ba6f05f87309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682416568059 2023-04-25 09:56:10,666 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ea71625a36a64f39822b2e80ce27b17b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1682416568716 2023-04-25 09:56:10,666 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting d2cd3db7f8114993a37c591a0c3148fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1682416569435 2023-04-25 09:56:10,678 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#518 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:10,690 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/b745ef8a970d425995a5c864dccf8dc4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b745ef8a970d425995a5c864dccf8dc4 2023-04-25 09:56:10,695 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into b745ef8a970d425995a5c864dccf8dc4(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:10,695 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:10,695 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=10, startTime=1682416570618; duration=0sec 2023-04-25 09:56:10,695 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:10,733 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,736 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,738 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,739 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,739 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,739 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,739 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-25 09:56:10,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:10,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:10,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:10,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:10,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:10,740 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:10,749 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/f157eabc232042d5857a388fef1197de 2023-04-25 09:56:10,761 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/4273aaccbaf649e18e8cdc9dfa407401 2023-04-25 09:56:10,774 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/bf7c9ca9f0074803969e2d59668ae6c6 2023-04-25 09:56:10,777 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/f157eabc232042d5857a388fef1197de as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f157eabc232042d5857a388fef1197de 2023-04-25 09:56:10,792 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f157eabc232042d5857a388fef1197de, entries=150, sequenceid=627, filesize=11.9 K 2023-04-25 09:56:10,793 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/4273aaccbaf649e18e8cdc9dfa407401 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/4273aaccbaf649e18e8cdc9dfa407401 2023-04-25 09:56:10,798 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/4273aaccbaf649e18e8cdc9dfa407401, entries=150, sequenceid=627, filesize=11.9 K 2023-04-25 09:56:10,799 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/bf7c9ca9f0074803969e2d59668ae6c6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bf7c9ca9f0074803969e2d59668ae6c6 2023-04-25 09:56:10,803 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bf7c9ca9f0074803969e2d59668ae6c6, entries=150, sequenceid=627, filesize=11.9 K 2023-04-25 09:56:10,803 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 6b4247a1e1c946a4543dfbc0edb52631 in 64ms, sequenceid=627, compaction requested=false 2023-04-25 09:56:10,803 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:10,803 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,803 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,805 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,805 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@430c189f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,811 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,813 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,815 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,815 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,823 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,823 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,924 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:10,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:10,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:10,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:10,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:10,927 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:10,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:10,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:10,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,927 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,929 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:10,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:10,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:10,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:10,932 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:10,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:10,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:10,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:10,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:10,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:10,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:10,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:10,939 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:10,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:10,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:10,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:10,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:10,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:10,943 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,943 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,943 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,943 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:10,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2405ccb2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:10,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:10,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,946 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:10,946 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:10,946 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:10,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:10,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2a56379c0ab848e9878df1fdf80c5d72 2023-04-25 09:56:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:10,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:10,953 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:10,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:10,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:10,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:10,953 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:10,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,959 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,959 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,959 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:10,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,959 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:10,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/5eba3a7b0c5645c59495306725397e84 2023-04-25 09:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:10,963 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:10,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:10,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:10,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,964 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:10,964 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:10,964 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,964 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:10,964 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:10,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:10,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:10,964 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:10,964 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:10,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:10,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:10,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/e82225e796544dc1a7ee4d99156103a9 2023-04-25 09:56:10,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/2a56379c0ab848e9878df1fdf80c5d72 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a56379c0ab848e9878df1fdf80c5d72 2023-04-25 09:56:10,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a56379c0ab848e9878df1fdf80c5d72, entries=300, sequenceid=637, filesize=19.0 K 2023-04-25 09:56:10,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/5eba3a7b0c5645c59495306725397e84 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5eba3a7b0c5645c59495306725397e84 2023-04-25 09:56:10,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5eba3a7b0c5645c59495306725397e84, entries=150, sequenceid=637, filesize=11.9 K 2023-04-25 09:56:10,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/e82225e796544dc1a7ee4d99156103a9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e82225e796544dc1a7ee4d99156103a9 2023-04-25 09:56:10,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e82225e796544dc1a7ee4d99156103a9, entries=150, sequenceid=637, filesize=11.9 K 2023-04-25 09:56:10,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 6b4247a1e1c946a4543dfbc0edb52631 in 55ms, sequenceid=637, compaction requested=true 2023-04-25 09:56:10,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:10,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:10,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:10,987 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:10,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:56:10,988 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 45036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:10,988 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:10,988 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:10,988 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/9318f3466f9748a580e6d74cf491d9bc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f157eabc232042d5857a388fef1197de, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a56379c0ab848e9878df1fdf80c5d72] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=44.0 K 2023-04-25 09:56:10,988 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9318f3466f9748a580e6d74cf491d9bc, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1682416569435 2023-04-25 09:56:10,989 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f157eabc232042d5857a388fef1197de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682416569778 2023-04-25 09:56:10,989 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2a56379c0ab848e9878df1fdf80c5d72, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1682416570918 2023-04-25 09:56:10,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=114.05 KB heapSize=339.66 KB 2023-04-25 09:56:10,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:10,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:10,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:10,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:10,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:10,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:10,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:10,996 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:11,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416631019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416631020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416631021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,054 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/5df3bcb99ca940d69de3d559a2e84f4a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5df3bcb99ca940d69de3d559a2e84f4a 2023-04-25 09:56:11,058 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 5df3bcb99ca940d69de3d559a2e84f4a(size=13.0 K), total size for store is 36.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:11,058 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:11,058 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=10, startTime=1682416570618; duration=0sec 2023-04-25 09:56:11,058 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:56:11,058 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:11,059 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:11,059 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:11,059 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,060 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b745ef8a970d425995a5c864dccf8dc4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bf7c9ca9f0074803969e2d59668ae6c6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e82225e796544dc1a7ee4d99156103a9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=36.8 K 2023-04-25 09:56:11,060 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b745ef8a970d425995a5c864dccf8dc4, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1682416569435 2023-04-25 09:56:11,060 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bf7c9ca9f0074803969e2d59668ae6c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682416569778 2023-04-25 09:56:11,060 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e82225e796544dc1a7ee4d99156103a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1682416570923 2023-04-25 09:56:11,066 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:11,068 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,076 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,077 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,080 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,080 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,083 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,083 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,084 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd1f2fa[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,089 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,091 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,091 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,093 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,093 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,093 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,096 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,098 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,098 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416631128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416631128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416631128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,198 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,202 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,204 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,206 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,206 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,207 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,207 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,208 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,208 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,208 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,209 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be8e9bc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,217 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,219 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,220 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,221 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,221 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,221 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,228 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,228 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,328 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,331 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,331 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,331 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,331 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416631332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416631333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,334 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,334 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,336 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416631333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,339 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7883dabe[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,342 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,344 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,347 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,347 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,354 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,354 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,418 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/8a7e53cb535b42939f89f014b56c4490 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8a7e53cb535b42939f89f014b56c4490 2023-04-25 09:56:11,422 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 8a7e53cb535b42939f89f014b56c4490(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:11,422 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:11,422 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=13, startTime=1682416570987; duration=0sec 2023-04-25 09:56:11,422 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:11,422 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:11,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/47d65fdd87664d7d939ad70116328130 2023-04-25 09:56:11,424 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:11,424 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:11,424 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,425 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5df3bcb99ca940d69de3d559a2e84f4a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/4273aaccbaf649e18e8cdc9dfa407401, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5eba3a7b0c5645c59495306725397e84] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=36.8 K 2023-04-25 09:56:11,425 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5df3bcb99ca940d69de3d559a2e84f4a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1682416569435 2023-04-25 09:56:11,426 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4273aaccbaf649e18e8cdc9dfa407401, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682416569778 2023-04-25 09:56:11,428 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 5eba3a7b0c5645c59495306725397e84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1682416570923 2023-04-25 09:56:11,434 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:11,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/36ef9b8164484ae19819626d7fbc6f46 2023-04-25 09:56:11,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/638ba462ab00447cb80515248b865983 2023-04-25 09:56:11,454 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,458 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/47d65fdd87664d7d939ad70116328130 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/47d65fdd87664d7d939ad70116328130 2023-04-25 09:56:11,459 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,463 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/47d65fdd87664d7d939ad70116328130, entries=250, sequenceid=658, filesize=16.7 K 2023-04-25 09:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/36ef9b8164484ae19819626d7fbc6f46 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36ef9b8164484ae19819626d7fbc6f46 2023-04-25 09:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e9e940[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,465 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,467 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36ef9b8164484ae19819626d7fbc6f46, entries=150, sequenceid=658, filesize=11.9 K 2023-04-25 09:56:11,471 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/638ba462ab00447cb80515248b865983 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/638ba462ab00447cb80515248b865983 2023-04-25 09:56:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,474 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,474 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/638ba462ab00447cb80515248b865983, entries=150, sequenceid=658, filesize=11.9 K 2023-04-25 09:56:11,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 6b4247a1e1c946a4543dfbc0edb52631 in 483ms, sequenceid=658, compaction requested=false 2023-04-25 09:56:11,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:11,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,480 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,481 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:11,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,482 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/4f1e24a823584c13953fa518f4ac6c68 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4f1e24a823584c13953fa518f4ac6c68 2023-04-25 09:56:11,485 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 4f1e24a823584c13953fa518f4ac6c68(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:11,485 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:11,485 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=13, startTime=1682416570987; duration=0sec 2023-04-25 09:56:11,485 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:11,581 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,584 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,588 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,589 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:11,589 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:11,596 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/005500687a1d4b4bad21e417d0cca585 2023-04-25 09:56:11,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:11,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,608 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/f97a8cca51284b86a07b65712d861e2e 2023-04-25 09:56:11,621 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/512a21c762f346c49928d4f822e518b9 2023-04-25 09:56:11,625 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/005500687a1d4b4bad21e417d0cca585 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/005500687a1d4b4bad21e417d0cca585 2023-04-25 09:56:11,628 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/005500687a1d4b4bad21e417d0cca585, entries=100, sequenceid=672, filesize=9.5 K 2023-04-25 09:56:11,629 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/f97a8cca51284b86a07b65712d861e2e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/f97a8cca51284b86a07b65712d861e2e 2023-04-25 09:56:11,632 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/f97a8cca51284b86a07b65712d861e2e, entries=100, sequenceid=672, filesize=9.5 K 2023-04-25 09:56:11,633 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/512a21c762f346c49928d4f822e518b9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/512a21c762f346c49928d4f822e518b9 2023-04-25 09:56:11,636 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/512a21c762f346c49928d4f822e518b9, entries=100, sequenceid=672, filesize=9.5 K 2023-04-25 09:56:11,637 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=33.54 KB/34350 for 6b4247a1e1c946a4543dfbc0edb52631 in 48ms, sequenceid=672, compaction requested=true 2023-04-25 09:56:11,637 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:11,637 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,637 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,640 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,640 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:11,641 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:11,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:11,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:11,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:11,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702d83c5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,643 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,643 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,644 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,644 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,644 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,645 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,645 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,645 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,645 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,649 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,649 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,651 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,651 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,651 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,651 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,653 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:11,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,658 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,658 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,659 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,659 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,659 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416631674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682416571659, max timestamp=1682416571677Num uniques 859; 2023-04-25 09:56:11,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 09:56:11,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682416571659, max timestamp=1682416571677Num uniques 859; 2023-04-25 09:56:11,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-25 09:56:11,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416631693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416631694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416631695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,759 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,763 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,763 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,765 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,766 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,766 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,766 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,766 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,766 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,767 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,768 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,768 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c143175[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,770 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,771 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,771 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,771 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,771 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,779 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,781 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,781 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,781 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,782 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,783 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,783 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,784 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416631780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,787 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:11,787 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,787 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,788 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416631801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416631803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416631803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,847 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/0fd821a2e2f44090b132c033f8fc02ea as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0fd821a2e2f44090b132c033f8fc02ea 2023-04-25 09:56:11,851 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 0fd821a2e2f44090b132c033f8fc02ea(size=13.1 K), total size for store is 34.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:11,851 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:11,851 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=13, startTime=1682416570987; duration=0sec 2023-04-25 09:56:11,851 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:11,887 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:11,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:11,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:11,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:11,890 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:11,890 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:11,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:11,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,891 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,892 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,892 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,893 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:11,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:11,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:11,894 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,894 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:11,894 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:11,894 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:11,894 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:11,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,897 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:11,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a244063[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:11,899 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:11,899 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,900 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:11,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:11,900 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,900 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:11,902 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,902 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:11,902 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,902 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:11,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:11,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:11,905 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:11,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:11,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:11,905 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:11,905 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:11,906 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,906 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,906 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:11,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,907 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:11,908 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:11,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:11,912 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,912 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,912 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:11,912 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:11,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:11,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:11,912 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:11,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:11,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:11,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:11,913 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:11,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:11,913 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:11,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:11,913 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:11,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:11,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:11,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416631989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,013 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:12,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416632009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416632010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416632011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:12,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:12,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:12,017 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:12,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:12,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:12,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,017 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,018 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,018 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,019 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:12,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:12,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:12,020 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:12,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:12,020 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:12,021 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:12,022 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,022 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:12,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ba0b8e[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:12,024 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:12,024 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,025 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:12,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:12,025 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:12,028 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:12,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:12,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:12,031 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:12,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:12,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:12,031 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:12,031 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:12,033 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,033 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,033 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,033 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,035 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:12,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:12,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,041 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:12,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,041 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:12,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/387b529f184840949b6ad7b62526e87d 2023-04-25 09:56:12,142 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:12,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:12,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:12,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:12,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:12,145 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:12,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:12,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:12,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,146 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,148 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:12,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:12,150 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:12,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:12,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:12,151 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:12,153 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,153 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b159517[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:12,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:12,155 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,156 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:12,156 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,157 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:12,159 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:12,161 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:12,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:12,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:12,161 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:12,162 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:12,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,164 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,165 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,166 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:12,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:12,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,168 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,169 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,169 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:12,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:12,169 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:12,270 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:12,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:12,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:12,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:12,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:12,273 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:12,273 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,273 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:12,273 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:12,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,274 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,276 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:12,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:12,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:12,277 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,277 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:12,277 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:12,277 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,277 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:12,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66478699[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:12,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:12,281 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,282 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:12,283 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:12,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,283 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:12,284 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,284 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:12,287 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:12,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:12,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:12,287 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:12,287 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:12,288 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,289 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,289 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,294 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:12,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:12,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:12,294 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,295 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:12,295 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:12,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416632295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416632317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416632318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416632318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,395 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:12,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:12,398 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:12,398 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,398 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:12,398 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:12,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,401 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,401 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:12,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:12,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:12,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:12,403 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,403 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:12,403 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:12,403 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:12,404 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:12,406 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:12,406 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e48f56a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:12,407 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:12,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:12,407 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:12,407 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,407 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:12,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:12,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:12,409 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:12,411 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:12,411 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:12,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:12,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:12,412 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:12,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:12,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:12,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:12,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:12,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:12,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:12,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,419 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:12,419 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:12,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:12,419 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:12,420 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/1bfb739dc9b94f248c7bae9aa13932f6 2023-04-25 09:56:12,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/2ea4bcd48f754ff5a12f2bd7a59666f8 2023-04-25 09:56:12,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/387b529f184840949b6ad7b62526e87d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/387b529f184840949b6ad7b62526e87d 2023-04-25 09:56:12,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/387b529f184840949b6ad7b62526e87d, entries=200, sequenceid=683, filesize=14.3 K 2023-04-25 09:56:12,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/1bfb739dc9b94f248c7bae9aa13932f6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/1bfb739dc9b94f248c7bae9aa13932f6 2023-04-25 09:56:12,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/1bfb739dc9b94f248c7bae9aa13932f6, entries=100, sequenceid=683, filesize=9.5 K 2023-04-25 09:56:12,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/2ea4bcd48f754ff5a12f2bd7a59666f8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/2ea4bcd48f754ff5a12f2bd7a59666f8 2023-04-25 09:56:12,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/2ea4bcd48f754ff5a12f2bd7a59666f8, entries=100, sequenceid=683, filesize=9.5 K 2023-04-25 09:56:12,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6b4247a1e1c946a4543dfbc0edb52631 in 861ms, sequenceid=683, compaction requested=true 2023-04-25 09:56:12,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:12,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:12,502 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:12,502 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:12,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:12,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:12,503 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54876 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:12,503 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45116 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:12,503 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:12,503 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:12,503 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,503 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,503 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8a7e53cb535b42939f89f014b56c4490, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/47d65fdd87664d7d939ad70116328130, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/005500687a1d4b4bad21e417d0cca585, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/387b529f184840949b6ad7b62526e87d] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=53.6 K 2023-04-25 09:56:12,503 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0fd821a2e2f44090b132c033f8fc02ea, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36ef9b8164484ae19819626d7fbc6f46, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/f97a8cca51284b86a07b65712d861e2e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/1bfb739dc9b94f248c7bae9aa13932f6] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=44.1 K 2023-04-25 09:56:12,504 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8a7e53cb535b42939f89f014b56c4490, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1682416570923 2023-04-25 09:56:12,504 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0fd821a2e2f44090b132c033f8fc02ea, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1682416570923 2023-04-25 09:56:12,504 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 47d65fdd87664d7d939ad70116328130, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682416570977 2023-04-25 09:56:12,504 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 36ef9b8164484ae19819626d7fbc6f46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682416570978 2023-04-25 09:56:12,505 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 005500687a1d4b4bad21e417d0cca585, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682416571019 2023-04-25 09:56:12,505 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f97a8cca51284b86a07b65712d861e2e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682416571019 2023-04-25 09:56:12,505 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 387b529f184840949b6ad7b62526e87d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682416571597 2023-04-25 09:56:12,505 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1bfb739dc9b94f248c7bae9aa13932f6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682416571597 2023-04-25 09:56:12,514 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#537 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:12,514 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#538 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:12,520 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:12,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:12,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:12,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:12,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:12,532 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:12,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:12,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:12,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,533 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,539 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:12,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,539 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:12,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:12,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:12,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:12,542 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,542 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:12,542 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=127.47 KB heapSize=354.41 KB 2023-04-25 09:56:12,543 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:12,543 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:12,543 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:12,543 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:12,543 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:12,543 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:12,546 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/4ddb00f60e8a4354bb3d7f3475ef7089 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/4ddb00f60e8a4354bb3d7f3475ef7089 2023-04-25 09:56:12,550 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into 4ddb00f60e8a4354bb3d7f3475ef7089(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:12,551 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:12,551 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=12, startTime=1682416572502; duration=0sec 2023-04-25 09:56:12,551 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:12,551 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:12,552 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45116 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:12,552 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:12,552 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:12,552 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4f1e24a823584c13953fa518f4ac6c68, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/638ba462ab00447cb80515248b865983, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/512a21c762f346c49928d4f822e518b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/2ea4bcd48f754ff5a12f2bd7a59666f8] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=44.1 K 2023-04-25 09:56:12,553 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4f1e24a823584c13953fa518f4ac6c68, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1682416570923 2023-04-25 09:56:12,553 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 638ba462ab00447cb80515248b865983, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1682416570978 2023-04-25 09:56:12,554 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 512a21c762f346c49928d4f822e518b9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1682416571019 2023-04-25 09:56:12,554 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 2ea4bcd48f754ff5a12f2bd7a59666f8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682416571597 2023-04-25 09:56:12,562 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#540 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:12,566 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/10b3c54315be4966ad914817d66c5440 2023-04-25 09:56:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:12,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:12,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416632828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416632835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416632837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416632837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,928 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/d5b17d851d424568a77a91c2ab7d977e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d5b17d851d424568a77a91c2ab7d977e 2023-04-25 09:56:12,932 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into d5b17d851d424568a77a91c2ab7d977e(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:12,932 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:12,932 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=12, startTime=1682416572502; duration=0sec 2023-04-25 09:56:12,932 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:12,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54364 deadline: 1682416632938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54384 deadline: 1682416632938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54328 deadline: 1682416632944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:12,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54396 deadline: 1682416632946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:12,981 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/bc9b61243af7403bafe56c50ff0f8e82 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bc9b61243af7403bafe56c50ff0f8e82 2023-04-25 09:56:12,984 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into bc9b61243af7403bafe56c50ff0f8e82(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:12,984 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:12,985 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=12, startTime=1682416572502; duration=0sec 2023-04-25 09:56:12,985 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:12,996 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/554ed2dcf6154a209023427922b2cfca 2023-04-25 09:56:13,007 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/49e407a6da984280ba0b3044b5f07a9f 2023-04-25 09:56:13,022 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/10b3c54315be4966ad914817d66c5440 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/10b3c54315be4966ad914817d66c5440 2023-04-25 09:56:13,026 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/10b3c54315be4966ad914817d66c5440, entries=150, sequenceid=712, filesize=11.9 K 2023-04-25 09:56:13,027 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/554ed2dcf6154a209023427922b2cfca as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/554ed2dcf6154a209023427922b2cfca 2023-04-25 09:56:13,031 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/554ed2dcf6154a209023427922b2cfca, entries=150, sequenceid=712, filesize=11.9 K 2023-04-25 09:56:13,031 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/49e407a6da984280ba0b3044b5f07a9f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/49e407a6da984280ba0b3044b5f07a9f 2023-04-25 09:56:13,040 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/49e407a6da984280ba0b3044b5f07a9f, entries=150, sequenceid=712, filesize=11.9 K 2023-04-25 09:56:13,041 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~354.36 KB/362864, currentSize=53.67 KB/54960 for 6b4247a1e1c946a4543dfbc0edb52631 in 499ms, sequenceid=712, compaction requested=false 2023-04-25 09:56:13,042 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:13,042 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:13,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:13,049 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:13,049 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29496156[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:13,052 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:13,052 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:13,053 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,053 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:13,055 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:13,057 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:13,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:13,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:13,057 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:13,057 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:13,059 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,059 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:13,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,059 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:13,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:13,060 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,061 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,065 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:13,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:13,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:13,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:13,066 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:13,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:13,066 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:13,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:13,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:13,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 09:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:13,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:13,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/8371ff7efb8c45298f90d2bdcf3e1474 2023-04-25 09:56:13,166 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:13,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:13,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:13,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:13,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:13,169 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:13,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:13,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:13,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:13,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,169 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,171 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:13,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:13,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,171 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:13,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:13,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:13,172 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:13,172 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. started... 2023-04-25 09:56:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:13,172 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. as already flushing 2023-04-25 09:56:13,173 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:13,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:13,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:13,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:13,173 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:13,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,174 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:13,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,174 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef3e445[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:13,175 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:13,176 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:13,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,177 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:13,179 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:13,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:13,181 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:13,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:13,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:13,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:13,181 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:13,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:13,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:13,183 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:13,184 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:13,187 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:13,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:13,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:13,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:13,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:13,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:13,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:13,187 DEBUG [Thread-3211] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:13,187 DEBUG [Thread-3211] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:13,192 DEBUG [Thread-3212] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:60014 2023-04-25 09:56:13,192 DEBUG [Thread-3212] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,195 DEBUG [Thread-3214] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:60014 2023-04-25 09:56:13,195 DEBUG [Thread-3214] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,196 DEBUG [Thread-3216] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:60014 2023-04-25 09:56:13,196 DEBUG [Thread-3205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:60014 2023-04-25 09:56:13,196 DEBUG [Thread-3205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,196 DEBUG [Thread-3216] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,197 DEBUG [Thread-3201] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df63a4 to 127.0.0.1:60014 2023-04-25 09:56:13,197 DEBUG [Thread-3201] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,198 DEBUG [Thread-3209] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:60014 2023-04-25 09:56:13,198 DEBUG [Thread-3220] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:60014 2023-04-25 09:56:13,198 DEBUG [Thread-3209] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,198 DEBUG [Thread-3220] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,198 DEBUG [Thread-3203] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:60014 2023-04-25 09:56:13,198 DEBUG [Thread-3203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,201 DEBUG [Thread-3218] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:60014 2023-04-25 09:56:13,201 DEBUG [Thread-3218] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:13,288 DEBUG [Thread-3211] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:13,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:13,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/16bd17f7cee44ef1bae8e8c3e07df78a 2023-04-25 09:56:13,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/43595862a4b6440296d2a691cf571dc9 2023-04-25 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/8371ff7efb8c45298f90d2bdcf3e1474 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8371ff7efb8c45298f90d2bdcf3e1474 2023-04-25 09:56:13,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8371ff7efb8c45298f90d2bdcf3e1474, entries=200, sequenceid=727, filesize=14.3 K 2023-04-25 09:56:13,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/16bd17f7cee44ef1bae8e8c3e07df78a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/16bd17f7cee44ef1bae8e8c3e07df78a 2023-04-25 09:56:13,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/16bd17f7cee44ef1bae8e8c3e07df78a, entries=150, sequenceid=727, filesize=11.9 K 2023-04-25 09:56:13,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/43595862a4b6440296d2a691cf571dc9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/43595862a4b6440296d2a691cf571dc9 2023-04-25 09:56:13,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/43595862a4b6440296d2a691cf571dc9, entries=150, sequenceid=727, filesize=11.9 K 2023-04-25 09:56:13,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=100.63 KB/103050 for 6b4247a1e1c946a4543dfbc0edb52631 in 842ms, sequenceid=727, compaction requested=true 2023-04-25 09:56:13,986 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:13,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:13,986 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:13,986 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:13,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:13,987 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:13,987 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40394 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:13,987 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/A is initiating minor compaction (all files) 2023-04-25 09:56:13,987 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/A in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:13,987 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/4ddb00f60e8a4354bb3d7f3475ef7089, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/10b3c54315be4966ad914817d66c5440, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8371ff7efb8c45298f90d2bdcf3e1474] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=39.4 K 2023-04-25 09:56:13,988 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:13,988 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4ddb00f60e8a4354bb3d7f3475ef7089, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682416571019 2023-04-25 09:56:13,988 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/B is initiating minor compaction (all files) 2023-04-25 09:56:13,988 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/B in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:13,988 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d5b17d851d424568a77a91c2ab7d977e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/554ed2dcf6154a209023427922b2cfca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/16bd17f7cee44ef1bae8e8c3e07df78a] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=37.1 K 2023-04-25 09:56:13,988 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 10b3c54315be4966ad914817d66c5440, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1682416571692 2023-04-25 09:56:13,988 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d5b17d851d424568a77a91c2ab7d977e, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682416571019 2023-04-25 09:56:13,989 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8371ff7efb8c45298f90d2bdcf3e1474, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1682416572827 2023-04-25 09:56:13,989 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 554ed2dcf6154a209023427922b2cfca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1682416571692 2023-04-25 09:56:13,989 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 16bd17f7cee44ef1bae8e8c3e07df78a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1682416572827 2023-04-25 09:56:13,993 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#A#compaction#546 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:13,994 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#B#compaction#547 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:14,000 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/27f1a41e1c9841ad83f5115af8f58823 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/27f1a41e1c9841ad83f5115af8f58823 2023-04-25 09:56:14,004 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/B of 6b4247a1e1c946a4543dfbc0edb52631 into 27f1a41e1c9841ad83f5115af8f58823(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:14,004 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:14,004 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/B, priority=13, startTime=1682416573986; duration=0sec 2023-04-25 09:56:14,004 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:14,004 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:14,005 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:14,005 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): 6b4247a1e1c946a4543dfbc0edb52631/C is initiating minor compaction (all files) 2023-04-25 09:56:14,005 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b4247a1e1c946a4543dfbc0edb52631/C in TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:14,005 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bc9b61243af7403bafe56c50ff0f8e82, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/49e407a6da984280ba0b3044b5f07a9f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/43595862a4b6440296d2a691cf571dc9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp, totalSize=37.1 K 2023-04-25 09:56:14,006 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bc9b61243af7403bafe56c50ff0f8e82, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682416571019 2023-04-25 09:56:14,006 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 49e407a6da984280ba0b3044b5f07a9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1682416571692 2023-04-25 09:56:14,006 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 43595862a4b6440296d2a691cf571dc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1682416572827 2023-04-25 09:56:14,010 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b4247a1e1c946a4543dfbc0edb52631#C#compaction#548 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:14,401 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/ad7ffe70167249298023a3a0dfd4d081 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ad7ffe70167249298023a3a0dfd4d081 2023-04-25 09:56:14,405 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/A of 6b4247a1e1c946a4543dfbc0edb52631 into ad7ffe70167249298023a3a0dfd4d081(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:14,405 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:14,405 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/A, priority=13, startTime=1682416573986; duration=0sec 2023-04-25 09:56:14,405 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:14,416 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/52266c72e3174e1793f741ea4b0438db as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/52266c72e3174e1793f741ea4b0438db 2023-04-25 09:56:14,420 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b4247a1e1c946a4543dfbc0edb52631/C of 6b4247a1e1c946a4543dfbc0edb52631 into 52266c72e3174e1793f741ea4b0438db(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:14,420 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:14,420 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631., storeName=6b4247a1e1c946a4543dfbc0edb52631/C, priority=13, startTime=1682416573986; duration=0sec 2023-04-25 09:56:14,420 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:14,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:14,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b4247a1e1c946a4543dfbc0edb52631 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:56:14,432 DEBUG [Thread-3207] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:60014 2023-04-25 09:56:14,432 DEBUG [Thread-3207] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:14,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=A 2023-04-25 09:56:14,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 09:56:14,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 162 2023-04-25 09:56:14,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 171 2023-04-25 09:56:14,432 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1952 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5855 rows 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1966 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5891 rows 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1955 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5865 rows 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1947 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5840 rows 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(402): scanned 1955 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(403): verified 5858 rows 2023-04-25 09:56:14,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:14,433 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:56:14,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=B 2023-04-25 09:56:14,433 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681750df to 127.0.0.1:60014 2023-04-25 09:56:14,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:14,433 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:14,434 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b4247a1e1c946a4543dfbc0edb52631, store=C 2023-04-25 09:56:14,434 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:14,435 INFO [Listener at localhost/33705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 09:56:14,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 09:56:14,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:14,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 09:56:14,440 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416574440"}]},"ts":"1682416574440"} 2023-04-25 09:56:14,441 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 09:56:14,537 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 09:56:14,538 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, UNASSIGN}] 2023-04-25 09:56:14,539 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, UNASSIGN 2023-04-25 09:56:14,540 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=6b4247a1e1c946a4543dfbc0edb52631, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:14,540 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416574540"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416574540"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416574540"}]},"ts":"1682416574540"} 2023-04-25 09:56:14,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 09:56:14,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:56:14,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:14,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6b4247a1e1c946a4543dfbc0edb52631, disabling compactions & flushes 2023-04-25 09:56:14,693 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:14,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 09:56:14,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/8737879395c6421481610935ee432557 2023-04-25 09:56:15,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 09:56:15,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/118d9410a89548aeaf598699faa26d68 2023-04-25 09:56:15,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 09:56:15,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/99704a609a7b4c88a92fe15d96203ad2 2023-04-25 09:56:15,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/A/8737879395c6421481610935ee432557 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8737879395c6421481610935ee432557 2023-04-25 09:56:15,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8737879395c6421481610935ee432557, entries=150, sequenceid=749, filesize=11.9 K 2023-04-25 09:56:15,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/B/118d9410a89548aeaf598699faa26d68 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/118d9410a89548aeaf598699faa26d68 2023-04-25 09:56:15,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/118d9410a89548aeaf598699faa26d68, entries=150, sequenceid=749, filesize=11.9 K 2023-04-25 09:56:15,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/.tmp/C/99704a609a7b4c88a92fe15d96203ad2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/99704a609a7b4c88a92fe15d96203ad2 2023-04-25 09:56:15,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/99704a609a7b4c88a92fe15d96203ad2, entries=150, sequenceid=749, filesize=11.9 K 2023-04-25 09:56:15,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 6b4247a1e1c946a4543dfbc0edb52631 in 1243ms, sequenceid=749, compaction requested=false 2023-04-25 09:56:15,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:15,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:15,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:15,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. after waiting 0 ms 2023-04-25 09:56:15,675 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:15,676 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ee7a1bd0e39946b2a298843c0e2b2b41, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/bc731e713d844002bb3c95b39b59213e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/01e9bd13eacc4bc39a84465f516eb6a3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/66c75fb9e7d94bc2a1ccc6cdab1145f8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2e99ed48df8b47c99c505810510b9712, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df75588bbb9e43eebf587ec733adf085, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df9eb23168e34148adc7131a4b378045, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/aab3846a50ba4866a795f328e551f9ca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/517c8d0d5e0e49fbbd284d4f7fee8e9a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/98c7027bb22d4d399154e4b06e7101e0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2d7eaba66c834cefb4daea1e95206bcf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/53aecdeecf5c47689b05ac89c98c749f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b8e8b42ab2004cd7868f529c850d1b6c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b59368168cbd424a810c73fd245d5764, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7ec38f999f614856858340ed15484f72, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b3f391b239974f6dbbc2b839bcfb6bcf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/612b34724bc041f0b11a62555708e99e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/94db529db4b94da4acde962e9186fc81, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/24450e65bf7b451c911aaabee61b8620, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7b3383f2ce8c45fe80830a88442f812b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2aedf8654f7d4f12b36ecaa7e16c067c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0ee0fef583d4d899561eaad29daba65, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a6f3e737aa14d0abe1e21e94aa5d2cc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/1472cdfe0b2a4e37a56e899914d15ff8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0c9539fa76947e982c57bead439b4ad, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/cfd9a40a9e904afdac1b07c281e380da, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ea6e5e458e9a4b87b2f7b509972832ef, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/415ff5ce446b4036beb1ba8a4d203f5a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f0ebd472b00d46aea79e7496d60ca6b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0322923de2246e193b2840527279dc0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/87e697611cb6489fb4f1c1bd8b253762, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/3368acc508864d12b4b27a841366fd2e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f67490c9374d4e959daf035139f78c60, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/62ed0f6cbf244084aeaca67de1244c95, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/9318f3466f9748a580e6d74cf491d9bc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f157eabc232042d5857a388fef1197de, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a56379c0ab848e9878df1fdf80c5d72, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8a7e53cb535b42939f89f014b56c4490, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/47d65fdd87664d7d939ad70116328130, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/005500687a1d4b4bad21e417d0cca585, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/387b529f184840949b6ad7b62526e87d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/4ddb00f60e8a4354bb3d7f3475ef7089, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/10b3c54315be4966ad914817d66c5440, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8371ff7efb8c45298f90d2bdcf3e1474] to archive 2023-04-25 09:56:15,677 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:56:15,678 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ee7a1bd0e39946b2a298843c0e2b2b41 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ee7a1bd0e39946b2a298843c0e2b2b41 2023-04-25 09:56:15,679 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/bc731e713d844002bb3c95b39b59213e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/bc731e713d844002bb3c95b39b59213e 2023-04-25 09:56:15,680 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/01e9bd13eacc4bc39a84465f516eb6a3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/01e9bd13eacc4bc39a84465f516eb6a3 2023-04-25 09:56:15,680 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/66c75fb9e7d94bc2a1ccc6cdab1145f8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/66c75fb9e7d94bc2a1ccc6cdab1145f8 2023-04-25 09:56:15,681 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2e99ed48df8b47c99c505810510b9712 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2e99ed48df8b47c99c505810510b9712 2023-04-25 09:56:15,682 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df75588bbb9e43eebf587ec733adf085 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df75588bbb9e43eebf587ec733adf085 2023-04-25 09:56:15,683 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df9eb23168e34148adc7131a4b378045 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/df9eb23168e34148adc7131a4b378045 2023-04-25 09:56:15,684 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/aab3846a50ba4866a795f328e551f9ca to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/aab3846a50ba4866a795f328e551f9ca 2023-04-25 09:56:15,684 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/517c8d0d5e0e49fbbd284d4f7fee8e9a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/517c8d0d5e0e49fbbd284d4f7fee8e9a 2023-04-25 09:56:15,685 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/98c7027bb22d4d399154e4b06e7101e0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/98c7027bb22d4d399154e4b06e7101e0 2023-04-25 09:56:15,686 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2d7eaba66c834cefb4daea1e95206bcf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2d7eaba66c834cefb4daea1e95206bcf 2023-04-25 09:56:15,687 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/53aecdeecf5c47689b05ac89c98c749f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/53aecdeecf5c47689b05ac89c98c749f 2023-04-25 09:56:15,688 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b8e8b42ab2004cd7868f529c850d1b6c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b8e8b42ab2004cd7868f529c850d1b6c 2023-04-25 09:56:15,689 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b59368168cbd424a810c73fd245d5764 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b59368168cbd424a810c73fd245d5764 2023-04-25 09:56:15,689 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7ec38f999f614856858340ed15484f72 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7ec38f999f614856858340ed15484f72 2023-04-25 09:56:15,690 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b3f391b239974f6dbbc2b839bcfb6bcf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/b3f391b239974f6dbbc2b839bcfb6bcf 2023-04-25 09:56:15,691 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/612b34724bc041f0b11a62555708e99e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/612b34724bc041f0b11a62555708e99e 2023-04-25 09:56:15,692 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/94db529db4b94da4acde962e9186fc81 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/94db529db4b94da4acde962e9186fc81 2023-04-25 09:56:15,692 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/24450e65bf7b451c911aaabee61b8620 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/24450e65bf7b451c911aaabee61b8620 2023-04-25 09:56:15,693 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7b3383f2ce8c45fe80830a88442f812b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/7b3383f2ce8c45fe80830a88442f812b 2023-04-25 09:56:15,694 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2aedf8654f7d4f12b36ecaa7e16c067c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2aedf8654f7d4f12b36ecaa7e16c067c 2023-04-25 09:56:15,695 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0ee0fef583d4d899561eaad29daba65 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0ee0fef583d4d899561eaad29daba65 2023-04-25 09:56:15,695 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a6f3e737aa14d0abe1e21e94aa5d2cc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a6f3e737aa14d0abe1e21e94aa5d2cc 2023-04-25 09:56:15,696 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/1472cdfe0b2a4e37a56e899914d15ff8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/1472cdfe0b2a4e37a56e899914d15ff8 2023-04-25 09:56:15,697 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0c9539fa76947e982c57bead439b4ad to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0c9539fa76947e982c57bead439b4ad 2023-04-25 09:56:15,697 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/cfd9a40a9e904afdac1b07c281e380da to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/cfd9a40a9e904afdac1b07c281e380da 2023-04-25 09:56:15,698 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ea6e5e458e9a4b87b2f7b509972832ef to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ea6e5e458e9a4b87b2f7b509972832ef 2023-04-25 09:56:15,699 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/415ff5ce446b4036beb1ba8a4d203f5a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/415ff5ce446b4036beb1ba8a4d203f5a 2023-04-25 09:56:15,699 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f0ebd472b00d46aea79e7496d60ca6b9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f0ebd472b00d46aea79e7496d60ca6b9 2023-04-25 09:56:15,700 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0322923de2246e193b2840527279dc0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/c0322923de2246e193b2840527279dc0 2023-04-25 09:56:15,701 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/87e697611cb6489fb4f1c1bd8b253762 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/87e697611cb6489fb4f1c1bd8b253762 2023-04-25 09:56:15,701 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/3368acc508864d12b4b27a841366fd2e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/3368acc508864d12b4b27a841366fd2e 2023-04-25 09:56:15,702 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f67490c9374d4e959daf035139f78c60 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f67490c9374d4e959daf035139f78c60 2023-04-25 09:56:15,703 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/62ed0f6cbf244084aeaca67de1244c95 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/62ed0f6cbf244084aeaca67de1244c95 2023-04-25 09:56:15,704 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/9318f3466f9748a580e6d74cf491d9bc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/9318f3466f9748a580e6d74cf491d9bc 2023-04-25 09:56:15,705 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f157eabc232042d5857a388fef1197de to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/f157eabc232042d5857a388fef1197de 2023-04-25 09:56:15,706 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a56379c0ab848e9878df1fdf80c5d72 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/2a56379c0ab848e9878df1fdf80c5d72 2023-04-25 09:56:15,707 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8a7e53cb535b42939f89f014b56c4490 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8a7e53cb535b42939f89f014b56c4490 2023-04-25 09:56:15,707 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/47d65fdd87664d7d939ad70116328130 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/47d65fdd87664d7d939ad70116328130 2023-04-25 09:56:15,708 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/005500687a1d4b4bad21e417d0cca585 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/005500687a1d4b4bad21e417d0cca585 2023-04-25 09:56:15,709 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/387b529f184840949b6ad7b62526e87d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/387b529f184840949b6ad7b62526e87d 2023-04-25 09:56:15,710 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/4ddb00f60e8a4354bb3d7f3475ef7089 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/4ddb00f60e8a4354bb3d7f3475ef7089 2023-04-25 09:56:15,711 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/10b3c54315be4966ad914817d66c5440 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/10b3c54315be4966ad914817d66c5440 2023-04-25 09:56:15,711 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8371ff7efb8c45298f90d2bdcf3e1474 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8371ff7efb8c45298f90d2bdcf3e1474 2023-04-25 09:56:15,714 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/38e21ab765214114952a95381d2b1bf7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/433d61aacea047ba8a20807228f54d74, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/b7322c1c2a5c451d95ecf223277b38ac, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0860af2586474602b5ff68031415ec51, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/577f3eca1bda433099da2c981adc776e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0865fe1820164363853b1a844200ecd0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/266c1dc680064e089271004ab6711c77, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/994f0a2a66594446b0aff4e7238e826d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6079a4a9f2944ecf8f46050ae59d0d26, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ee3ebec3547746f398e29bfaf43da5e4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/cdee0da979254bab955e5b55945651bd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9ee3178729cd4be48c6fc72521bb248e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/86581cf452854997ab3c35e97c6a97d0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6a7f07ef4fc04d34860bff301833cce8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/353fc21a742c4547bcbc88cbf1b936c6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7adc490af75e49e3b3f613772a6f0e3a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/898569fe91274af4b79bdd6d7a1a9417, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36f48327371d44f4b838ae6666641890, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0ce68494b7ac4e2a8fa855d08205539f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/60e129145e10419fa15e12dcb7bc3917, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/e8dc37c5b4c14ed4acb087251b3ee4c0, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/da4ffc59dffb43dea0672567271fb769, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/65613ed82ca3499688cf2a1164809adf, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/655c8fd156d443748c8c70b0d50e0866, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ca90b95d6daf4ba8960870a9bad7c0f7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9f7c6c58640345c8ac9e71b0cd102ad4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d1dca1c477d04c6aba744376664dd4bc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/29268f33f31f41f0be35491c72bd5c31, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7d6db2dd62284d77a798a39c0b3eb801, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/2daa6869b77848bcb624941edae47a37, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/46ffd14609024ced86b825ea234300bb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/852088a3a96b47e98fba8bae1d6f94b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ccc0b01e8362402394cc93bbfd35e861, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5df3bcb99ca940d69de3d559a2e84f4a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/475f234febc54baf9b5042a391b8a1f5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/4273aaccbaf649e18e8cdc9dfa407401, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0fd821a2e2f44090b132c033f8fc02ea, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5eba3a7b0c5645c59495306725397e84, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36ef9b8164484ae19819626d7fbc6f46, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/f97a8cca51284b86a07b65712d861e2e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d5b17d851d424568a77a91c2ab7d977e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/1bfb739dc9b94f248c7bae9aa13932f6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/554ed2dcf6154a209023427922b2cfca, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/16bd17f7cee44ef1bae8e8c3e07df78a] to archive 2023-04-25 09:56:15,715 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:56:15,716 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/38e21ab765214114952a95381d2b1bf7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/38e21ab765214114952a95381d2b1bf7 2023-04-25 09:56:15,717 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/433d61aacea047ba8a20807228f54d74 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/433d61aacea047ba8a20807228f54d74 2023-04-25 09:56:15,718 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/b7322c1c2a5c451d95ecf223277b38ac to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/b7322c1c2a5c451d95ecf223277b38ac 2023-04-25 09:56:15,718 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0860af2586474602b5ff68031415ec51 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0860af2586474602b5ff68031415ec51 2023-04-25 09:56:15,719 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/577f3eca1bda433099da2c981adc776e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/577f3eca1bda433099da2c981adc776e 2023-04-25 09:56:15,720 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0865fe1820164363853b1a844200ecd0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0865fe1820164363853b1a844200ecd0 2023-04-25 09:56:15,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/266c1dc680064e089271004ab6711c77 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/266c1dc680064e089271004ab6711c77 2023-04-25 09:56:15,721 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/994f0a2a66594446b0aff4e7238e826d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/994f0a2a66594446b0aff4e7238e826d 2023-04-25 09:56:15,722 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6079a4a9f2944ecf8f46050ae59d0d26 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6079a4a9f2944ecf8f46050ae59d0d26 2023-04-25 09:56:15,723 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ee3ebec3547746f398e29bfaf43da5e4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ee3ebec3547746f398e29bfaf43da5e4 2023-04-25 09:56:15,723 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/cdee0da979254bab955e5b55945651bd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/cdee0da979254bab955e5b55945651bd 2023-04-25 09:56:15,724 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9ee3178729cd4be48c6fc72521bb248e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9ee3178729cd4be48c6fc72521bb248e 2023-04-25 09:56:15,725 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/86581cf452854997ab3c35e97c6a97d0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/86581cf452854997ab3c35e97c6a97d0 2023-04-25 09:56:15,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6a7f07ef4fc04d34860bff301833cce8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/6a7f07ef4fc04d34860bff301833cce8 2023-04-25 09:56:15,726 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/353fc21a742c4547bcbc88cbf1b936c6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/353fc21a742c4547bcbc88cbf1b936c6 2023-04-25 09:56:15,727 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7adc490af75e49e3b3f613772a6f0e3a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7adc490af75e49e3b3f613772a6f0e3a 2023-04-25 09:56:15,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/898569fe91274af4b79bdd6d7a1a9417 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/898569fe91274af4b79bdd6d7a1a9417 2023-04-25 09:56:15,728 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36f48327371d44f4b838ae6666641890 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36f48327371d44f4b838ae6666641890 2023-04-25 09:56:15,729 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0ce68494b7ac4e2a8fa855d08205539f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0ce68494b7ac4e2a8fa855d08205539f 2023-04-25 09:56:15,730 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/60e129145e10419fa15e12dcb7bc3917 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/60e129145e10419fa15e12dcb7bc3917 2023-04-25 09:56:15,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/e8dc37c5b4c14ed4acb087251b3ee4c0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/e8dc37c5b4c14ed4acb087251b3ee4c0 2023-04-25 09:56:15,731 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/da4ffc59dffb43dea0672567271fb769 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/da4ffc59dffb43dea0672567271fb769 2023-04-25 09:56:15,732 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/65613ed82ca3499688cf2a1164809adf to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/65613ed82ca3499688cf2a1164809adf 2023-04-25 09:56:15,733 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/655c8fd156d443748c8c70b0d50e0866 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/655c8fd156d443748c8c70b0d50e0866 2023-04-25 09:56:15,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ca90b95d6daf4ba8960870a9bad7c0f7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ca90b95d6daf4ba8960870a9bad7c0f7 2023-04-25 09:56:15,734 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9f7c6c58640345c8ac9e71b0cd102ad4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/9f7c6c58640345c8ac9e71b0cd102ad4 2023-04-25 09:56:15,735 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d1dca1c477d04c6aba744376664dd4bc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d1dca1c477d04c6aba744376664dd4bc 2023-04-25 09:56:15,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/29268f33f31f41f0be35491c72bd5c31 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/29268f33f31f41f0be35491c72bd5c31 2023-04-25 09:56:15,736 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7d6db2dd62284d77a798a39c0b3eb801 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/7d6db2dd62284d77a798a39c0b3eb801 2023-04-25 09:56:15,737 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/2daa6869b77848bcb624941edae47a37 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/2daa6869b77848bcb624941edae47a37 2023-04-25 09:56:15,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/46ffd14609024ced86b825ea234300bb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/46ffd14609024ced86b825ea234300bb 2023-04-25 09:56:15,738 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/852088a3a96b47e98fba8bae1d6f94b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/852088a3a96b47e98fba8bae1d6f94b1 2023-04-25 09:56:15,739 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ccc0b01e8362402394cc93bbfd35e861 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/ccc0b01e8362402394cc93bbfd35e861 2023-04-25 09:56:15,740 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5df3bcb99ca940d69de3d559a2e84f4a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5df3bcb99ca940d69de3d559a2e84f4a 2023-04-25 09:56:15,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/475f234febc54baf9b5042a391b8a1f5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/475f234febc54baf9b5042a391b8a1f5 2023-04-25 09:56:15,741 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/4273aaccbaf649e18e8cdc9dfa407401 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/4273aaccbaf649e18e8cdc9dfa407401 2023-04-25 09:56:15,742 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0fd821a2e2f44090b132c033f8fc02ea to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/0fd821a2e2f44090b132c033f8fc02ea 2023-04-25 09:56:15,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5eba3a7b0c5645c59495306725397e84 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/5eba3a7b0c5645c59495306725397e84 2023-04-25 09:56:15,743 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36ef9b8164484ae19819626d7fbc6f46 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/36ef9b8164484ae19819626d7fbc6f46 2023-04-25 09:56:15,744 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/f97a8cca51284b86a07b65712d861e2e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/f97a8cca51284b86a07b65712d861e2e 2023-04-25 09:56:15,745 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d5b17d851d424568a77a91c2ab7d977e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/d5b17d851d424568a77a91c2ab7d977e 2023-04-25 09:56:15,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/1bfb739dc9b94f248c7bae9aa13932f6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/1bfb739dc9b94f248c7bae9aa13932f6 2023-04-25 09:56:15,746 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/554ed2dcf6154a209023427922b2cfca to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/554ed2dcf6154a209023427922b2cfca 2023-04-25 09:56:15,747 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/16bd17f7cee44ef1bae8e8c3e07df78a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/16bd17f7cee44ef1bae8e8c3e07df78a 2023-04-25 09:56:15,750 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d20c015660da4d35ae165a46e42a12f8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4d1ed629120541a0a31104e033ec21c8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3498732266c94a498fd5f2eef45b1415, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/912fd657c28145ddbdd31bb61130fb4d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/750854fcd28d4fd686ca449f00100f42, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fd91c098a2c24520abc807be736f0a5b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/9dd88a6fdc0c422db0bdf7bcf1e25c82, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/6bbaf1f0d5bb4e38829ca819d44008c6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/c80b413f3da448d5be1477b46baaa302, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/73648a6c0f7c498ca57f5949ec9296b6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/5e44fb879bce4660bf0061e313a54f46, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/0dbcee85358b4934b604df7226a8cc71, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d7a0e7aa72fc434ea74092c6bc297a56, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/984efee93dc84de2b6139c526710eec3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3530c8f9348241aeb8b920d85dfb2238, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e806777aac174a1f8361b7ddc1238cbb, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e0085c70a11f4f9285d326b95426cf17, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/abc7c4b75c004b59889d3189166bc875, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3ce90875c72f41349e68c0dbdea368f9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/617b93ddbafa44bc8095cc64ac4b0914, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/23de689335f84c2a9bbf9476ba8289cd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fb832c3cc13d46bbb550e44a5cbcbd52, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/50238a0a0b5543db8346d71ca7692e04, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/892c23eb428c4a9da11393fabf8b4b47, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b7d0a70a22fa40bdad217dd7c1d0c53f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/07bd034e0a4448dba97f852451219300, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/22965ddafe2d44e482ace90c437e0811, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/63298f890e584945a3e4b9f07b627baa, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/552d9ef420ae485abc12edd1bcc098dd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/599ee509772c4e2fa7bdad46afc6371f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3bd25b11525341dd9f07025ba9696416, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3d8c8b2f7529452c9476ba6f05f87309, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/ea71625a36a64f39822b2e80ce27b17b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b745ef8a970d425995a5c864dccf8dc4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d2cd3db7f8114993a37c591a0c3148fc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bf7c9ca9f0074803969e2d59668ae6c6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4f1e24a823584c13953fa518f4ac6c68, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e82225e796544dc1a7ee4d99156103a9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/638ba462ab00447cb80515248b865983, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/512a21c762f346c49928d4f822e518b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bc9b61243af7403bafe56c50ff0f8e82, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/2ea4bcd48f754ff5a12f2bd7a59666f8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/49e407a6da984280ba0b3044b5f07a9f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/43595862a4b6440296d2a691cf571dc9] to archive 2023-04-25 09:56:15,751 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:56:15,752 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d20c015660da4d35ae165a46e42a12f8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d20c015660da4d35ae165a46e42a12f8 2023-04-25 09:56:15,753 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4d1ed629120541a0a31104e033ec21c8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4d1ed629120541a0a31104e033ec21c8 2023-04-25 09:56:15,754 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3498732266c94a498fd5f2eef45b1415 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3498732266c94a498fd5f2eef45b1415 2023-04-25 09:56:15,755 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/912fd657c28145ddbdd31bb61130fb4d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/912fd657c28145ddbdd31bb61130fb4d 2023-04-25 09:56:15,756 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/750854fcd28d4fd686ca449f00100f42 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/750854fcd28d4fd686ca449f00100f42 2023-04-25 09:56:15,757 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fd91c098a2c24520abc807be736f0a5b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fd91c098a2c24520abc807be736f0a5b 2023-04-25 09:56:15,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/9dd88a6fdc0c422db0bdf7bcf1e25c82 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/9dd88a6fdc0c422db0bdf7bcf1e25c82 2023-04-25 09:56:15,758 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/6bbaf1f0d5bb4e38829ca819d44008c6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/6bbaf1f0d5bb4e38829ca819d44008c6 2023-04-25 09:56:15,759 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/c80b413f3da448d5be1477b46baaa302 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/c80b413f3da448d5be1477b46baaa302 2023-04-25 09:56:15,760 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/73648a6c0f7c498ca57f5949ec9296b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/73648a6c0f7c498ca57f5949ec9296b6 2023-04-25 09:56:15,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/5e44fb879bce4660bf0061e313a54f46 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/5e44fb879bce4660bf0061e313a54f46 2023-04-25 09:56:15,761 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/0dbcee85358b4934b604df7226a8cc71 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/0dbcee85358b4934b604df7226a8cc71 2023-04-25 09:56:15,762 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d7a0e7aa72fc434ea74092c6bc297a56 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d7a0e7aa72fc434ea74092c6bc297a56 2023-04-25 09:56:15,763 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/984efee93dc84de2b6139c526710eec3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/984efee93dc84de2b6139c526710eec3 2023-04-25 09:56:15,764 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3530c8f9348241aeb8b920d85dfb2238 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3530c8f9348241aeb8b920d85dfb2238 2023-04-25 09:56:15,764 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e806777aac174a1f8361b7ddc1238cbb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e806777aac174a1f8361b7ddc1238cbb 2023-04-25 09:56:15,765 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e0085c70a11f4f9285d326b95426cf17 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e0085c70a11f4f9285d326b95426cf17 2023-04-25 09:56:15,766 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/abc7c4b75c004b59889d3189166bc875 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/abc7c4b75c004b59889d3189166bc875 2023-04-25 09:56:15,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3ce90875c72f41349e68c0dbdea368f9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3ce90875c72f41349e68c0dbdea368f9 2023-04-25 09:56:15,767 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/617b93ddbafa44bc8095cc64ac4b0914 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/617b93ddbafa44bc8095cc64ac4b0914 2023-04-25 09:56:15,768 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/23de689335f84c2a9bbf9476ba8289cd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/23de689335f84c2a9bbf9476ba8289cd 2023-04-25 09:56:15,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fb832c3cc13d46bbb550e44a5cbcbd52 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/fb832c3cc13d46bbb550e44a5cbcbd52 2023-04-25 09:56:15,769 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/50238a0a0b5543db8346d71ca7692e04 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/50238a0a0b5543db8346d71ca7692e04 2023-04-25 09:56:15,770 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/892c23eb428c4a9da11393fabf8b4b47 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/892c23eb428c4a9da11393fabf8b4b47 2023-04-25 09:56:15,772 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b7d0a70a22fa40bdad217dd7c1d0c53f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b7d0a70a22fa40bdad217dd7c1d0c53f 2023-04-25 09:56:15,773 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/07bd034e0a4448dba97f852451219300 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/07bd034e0a4448dba97f852451219300 2023-04-25 09:56:15,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/22965ddafe2d44e482ace90c437e0811 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/22965ddafe2d44e482ace90c437e0811 2023-04-25 09:56:15,774 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/63298f890e584945a3e4b9f07b627baa to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/63298f890e584945a3e4b9f07b627baa 2023-04-25 09:56:15,775 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/552d9ef420ae485abc12edd1bcc098dd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/552d9ef420ae485abc12edd1bcc098dd 2023-04-25 09:56:15,776 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/599ee509772c4e2fa7bdad46afc6371f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/599ee509772c4e2fa7bdad46afc6371f 2023-04-25 09:56:15,777 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3bd25b11525341dd9f07025ba9696416 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3bd25b11525341dd9f07025ba9696416 2023-04-25 09:56:15,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3d8c8b2f7529452c9476ba6f05f87309 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/3d8c8b2f7529452c9476ba6f05f87309 2023-04-25 09:56:15,778 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/ea71625a36a64f39822b2e80ce27b17b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/ea71625a36a64f39822b2e80ce27b17b 2023-04-25 09:56:15,779 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b745ef8a970d425995a5c864dccf8dc4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/b745ef8a970d425995a5c864dccf8dc4 2023-04-25 09:56:15,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d2cd3db7f8114993a37c591a0c3148fc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/d2cd3db7f8114993a37c591a0c3148fc 2023-04-25 09:56:15,780 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bf7c9ca9f0074803969e2d59668ae6c6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bf7c9ca9f0074803969e2d59668ae6c6 2023-04-25 09:56:15,781 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4f1e24a823584c13953fa518f4ac6c68 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/4f1e24a823584c13953fa518f4ac6c68 2023-04-25 09:56:15,782 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e82225e796544dc1a7ee4d99156103a9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/e82225e796544dc1a7ee4d99156103a9 2023-04-25 09:56:15,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/638ba462ab00447cb80515248b865983 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/638ba462ab00447cb80515248b865983 2023-04-25 09:56:15,783 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/512a21c762f346c49928d4f822e518b9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/512a21c762f346c49928d4f822e518b9 2023-04-25 09:56:15,784 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bc9b61243af7403bafe56c50ff0f8e82 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/bc9b61243af7403bafe56c50ff0f8e82 2023-04-25 09:56:15,785 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/2ea4bcd48f754ff5a12f2bd7a59666f8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/2ea4bcd48f754ff5a12f2bd7a59666f8 2023-04-25 09:56:15,786 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/49e407a6da984280ba0b3044b5f07a9f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/49e407a6da984280ba0b3044b5f07a9f 2023-04-25 09:56:15,787 DEBUG [StoreCloser-TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/43595862a4b6440296d2a691cf571dc9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/43595862a4b6440296d2a691cf571dc9 2023-04-25 09:56:15,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/recovered.edits/752.seqid, newMaxSeqId=752, maxSeqId=1 2023-04-25 09:56:15,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631. 2023-04-25 09:56:15,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6b4247a1e1c946a4543dfbc0edb52631: 2023-04-25 09:56:15,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:15,794 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=6b4247a1e1c946a4543dfbc0edb52631, regionState=CLOSED 2023-04-25 09:56:15,794 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416575794"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416575794"}]},"ts":"1682416575794"} 2023-04-25 09:56:15,796 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-25 09:56:15,796 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 6b4247a1e1c946a4543dfbc0edb52631, server=jenkins-hbase4.apache.org,39407,1682416417123 in 1.2540 sec 2023-04-25 09:56:15,797 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-25 09:56:15,797 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b4247a1e1c946a4543dfbc0edb52631, UNASSIGN in 1.2580 sec 2023-04-25 09:56:15,797 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416575797"}]},"ts":"1682416575797"} 2023-04-25 09:56:15,798 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 09:56:15,799 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 09:56:15,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.3640 sec 2023-04-25 09:56:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-25 09:56:16,545 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-25 09:56:16,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 09:56:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,548 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-25 09:56:16,548 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,553 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:16,554 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/recovered.edits] 2023-04-25 09:56:16,556 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8737879395c6421481610935ee432557 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/8737879395c6421481610935ee432557 2023-04-25 09:56:16,557 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ad7ffe70167249298023a3a0dfd4d081 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/A/ad7ffe70167249298023a3a0dfd4d081 2023-04-25 09:56:16,559 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/118d9410a89548aeaf598699faa26d68 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/118d9410a89548aeaf598699faa26d68 2023-04-25 09:56:16,560 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/27f1a41e1c9841ad83f5115af8f58823 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/B/27f1a41e1c9841ad83f5115af8f58823 2023-04-25 09:56:16,562 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/52266c72e3174e1793f741ea4b0438db to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/52266c72e3174e1793f741ea4b0438db 2023-04-25 09:56:16,563 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/99704a609a7b4c88a92fe15d96203ad2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/C/99704a609a7b4c88a92fe15d96203ad2 2023-04-25 09:56:16,565 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/recovered.edits/752.seqid to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631/recovered.edits/752.seqid 2023-04-25 09:56:16,565 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/6b4247a1e1c946a4543dfbc0edb52631 2023-04-25 09:56:16,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:56:16,567 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,568 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 09:56:16,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 09:56:16,571 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,571 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 09:56:16,571 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682416576571"}]},"ts":"9223372036854775807"} 2023-04-25 09:56:16,572 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 09:56:16,572 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6b4247a1e1c946a4543dfbc0edb52631, NAME => 'TestAcidGuarantees,,1682416551009.6b4247a1e1c946a4543dfbc0edb52631.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 09:56:16,572 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 09:56:16,572 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682416576572"}]},"ts":"9223372036854775807"} 2023-04-25 09:56:16,573 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 09:56:16,575 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 30 msec 2023-04-25 09:56:16,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-25 09:56:16,649 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-25 09:56:16,661 INFO [Listener at localhost/33705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=231 (was 228) - Thread LEAK? -, OpenFileDescriptor=584 (was 533) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=397 (was 372) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=4527 (was 4409) - AvailableMemoryMB LEAK? - 2023-04-25 09:56:16,674 INFO [Listener at localhost/33705] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=231, OpenFileDescriptor=584, MaxFileDescriptor=60000, SystemLoadAverage=397, ProcessCount=171, AvailableMemoryMB=4526 2023-04-25 09:56:16,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:56:16,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:56:16,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:16,679 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-25 09:56:16,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-25 09:56:16,680 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-25 09:56:16,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 09:56:16,681 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:16,681 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 empty. 2023-04-25 09:56:16,681 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:16,681 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:56:16,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 09:56:16,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 09:56:17,088 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:56:17,089 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fb513dc5b3a62f2dba9b88a4e6de5394, NAME => 'TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp 2023-04-25 09:56:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 09:56:17,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:56:17,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fb513dc5b3a62f2dba9b88a4e6de5394, disabling compactions & flushes 2023-04-25 09:56:17,494 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. after waiting 0 ms 2023-04-25 09:56:17,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,494 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:17,496 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-25 09:56:17,497 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416577497"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416577497"}]},"ts":"1682416577497"} 2023-04-25 09:56:17,498 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-25 09:56:17,498 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-25 09:56:17,499 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416577498"}]},"ts":"1682416577498"} 2023-04-25 09:56:17,499 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-25 09:56:17,503 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, ASSIGN}] 2023-04-25 09:56:17,505 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, ASSIGN 2023-04-25 09:56:17,505 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=false 2023-04-25 09:56:17,657 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:17,657 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416577657"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416577657"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416577657"}]},"ts":"1682416577657"} 2023-04-25 09:56:17,658 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:56:17,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 09:56:17,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fb513dc5b3a62f2dba9b88a4e6de5394, NAME => 'TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:56:17,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:56:17,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,814 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,815 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A 2023-04-25 09:56:17,815 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A 2023-04-25 09:56:17,816 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:56:17,816 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb513dc5b3a62f2dba9b88a4e6de5394 columnFamilyName A 2023-04-25 09:56:17,816 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(310): Store=fb513dc5b3a62f2dba9b88a4e6de5394/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:56:17,816 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,817 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B 2023-04-25 09:56:17,817 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B 2023-04-25 09:56:17,818 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:56:17,818 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb513dc5b3a62f2dba9b88a4e6de5394 columnFamilyName B 2023-04-25 09:56:17,818 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(310): Store=fb513dc5b3a62f2dba9b88a4e6de5394/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:56:17,818 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,819 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C 2023-04-25 09:56:17,819 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C 2023-04-25 09:56:17,820 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:56:17,820 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb513dc5b3a62f2dba9b88a4e6de5394 columnFamilyName C 2023-04-25 09:56:17,820 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(310): Store=fb513dc5b3a62f2dba9b88a4e6de5394/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:56:17,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:56:17,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:17,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-25 09:56:17,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fb513dc5b3a62f2dba9b88a4e6de5394; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70173818, jitterRate=0.04567137360572815}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:56:17,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:17,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., pid=56, masterSystemTime=1682416577810 2023-04-25 09:56:17,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:17,828 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:17,828 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416577828"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416577828"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416577828"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416577828"}]},"ts":"1682416577828"} 2023-04-25 09:56:17,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-25 09:56:17,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 in 171 msec 2023-04-25 09:56:17,831 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-25 09:56:17,831 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, ASSIGN in 327 msec 2023-04-25 09:56:17,831 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-25 09:56:17,831 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416577831"}]},"ts":"1682416577831"} 2023-04-25 09:56:17,832 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-25 09:56:17,834 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-25 09:56:17,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-25 09:56:18,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-25 09:56:18,784 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-25 09:56:18,785 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x470742aa to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:18,789 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ced4ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:18,790 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:18,792 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:18,792 DEBUG [Listener at localhost/33705] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-25 09:56:18,794 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40444, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-25 09:56:18,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-25 09:56:18,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-25 09:56:18,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:19,205 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 09:56:19,206 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-25 09:56:19,207 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-25 09:56:19,208 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-25 09:56:19,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, REOPEN/MOVE}] 2023-04-25 09:56:19,212 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, REOPEN/MOVE 2023-04-25 09:56:19,213 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,213 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416579213"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416579213"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416579213"}]},"ts":"1682416579213"} 2023-04-25 09:56:19,214 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:56:19,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fb513dc5b3a62f2dba9b88a4e6de5394, disabling compactions & flushes 2023-04-25 09:56:19,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. after waiting 0 ms 2023-04-25 09:56:19,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-25 09:56:19,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:19,370 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: fb513dc5b3a62f2dba9b88a4e6de5394 to self. 2023-04-25 09:56:19,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,372 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=CLOSED 2023-04-25 09:56:19,372 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416579372"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416579372"}]},"ts":"1682416579372"} 2023-04-25 09:56:19,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-25 09:56:19,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 in 159 msec 2023-04-25 09:56:19,374 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,39407,1682416417123; forceNewPlan=false, retain=true 2023-04-25 09:56:19,525 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,525 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416579525"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416579525"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416579525"}]},"ts":"1682416579525"} 2023-04-25 09:56:19,526 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:56:19,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fb513dc5b3a62f2dba9b88a4e6de5394, NAME => 'TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.', STARTKEY => '', ENDKEY => ''} 2023-04-25 09:56:19,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-25 09:56:19,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,683 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,684 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A 2023-04-25 09:56:19,684 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A 2023-04-25 09:56:19,684 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:56:19,685 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb513dc5b3a62f2dba9b88a4e6de5394 columnFamilyName A 2023-04-25 09:56:19,686 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(310): Store=fb513dc5b3a62f2dba9b88a4e6de5394/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:56:19,686 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,687 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B 2023-04-25 09:56:19,687 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B 2023-04-25 09:56:19,687 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:56:19,687 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb513dc5b3a62f2dba9b88a4e6de5394 columnFamilyName B 2023-04-25 09:56:19,687 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(310): Store=fb513dc5b3a62f2dba9b88a4e6de5394/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:56:19,687 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,688 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C 2023-04-25 09:56:19,688 DEBUG [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C 2023-04-25 09:56:19,688 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-25 09:56:19,688 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb513dc5b3a62f2dba9b88a4e6de5394 columnFamilyName C 2023-04-25 09:56:19,689 INFO [StoreOpener-fb513dc5b3a62f2dba9b88a4e6de5394-1] regionserver.HStore(310): Store=fb513dc5b3a62f2dba9b88a4e6de5394/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-25 09:56:19,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-25 09:56:19,693 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fb513dc5b3a62f2dba9b88a4e6de5394; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71377065, jitterRate=0.06360115110874176}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-25 09:56:19,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:19,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., pid=61, masterSystemTime=1682416579677 2023-04-25 09:56:19,695 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,695 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,696 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,696 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416579696"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682416579696"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682416579696"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682416579696"}]},"ts":"1682416579696"} 2023-04-25 09:56:19,698 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-25 09:56:19,698 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 in 171 msec 2023-04-25 09:56:19,699 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-25 09:56:19,699 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, REOPEN/MOVE in 487 msec 2023-04-25 09:56:19,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-25 09:56:19,701 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-25 09:56:19,702 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-25 09:56:19,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-25 09:56:19,703 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,708 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,708 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,712 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,712 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,716 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,716 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,720 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,720 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,724 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,725 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,728 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,728 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,734 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,734 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,745 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,746 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,757 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,758 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:60014 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-25 09:56:19,762 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-25 09:56:19,773 DEBUG [hconnection-0x479c9d7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,773 DEBUG [hconnection-0x376e389a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,773 DEBUG [hconnection-0x7eb8bc38-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:19,775 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,776 DEBUG [hconnection-0x202b75f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,779 DEBUG [hconnection-0xe4d4baa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,779 DEBUG [hconnection-0x2dcdeab0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,780 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,780 DEBUG [hconnection-0x13ba9604-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,780 DEBUG [hconnection-0x31e649e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,782 DEBUG [hconnection-0x2e28fa00-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,782 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,782 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,784 DEBUG [hconnection-0x515e1278-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-25 09:56:19,784 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,786 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,792 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,792 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,793 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48676, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,793 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-25 09:56:19,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:19,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:19,806 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:19,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:19,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:19,806 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:19,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,807 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,808 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:19,808 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:19,809 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:19,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:19,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:19,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:19,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:19,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:19,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:19,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:19,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:19,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:19,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:19,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:19,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:19,812 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:19,816 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:19,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74dacc4c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:19,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:19,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,820 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:19,820 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:19,822 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:19,825 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:19,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:19,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:19,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:19,825 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:19,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416579818, max timestamp=1682416579825Num uniques 859; 2023-04-25 09:56:19,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:19,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416579819, max timestamp=1682416579825Num uniques 859; 2023-04-25 09:56:19,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:19,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:19,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416579819, max timestamp=1682416579825Num uniques 859; 2023-04-25 09:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:19,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:19,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:19,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,830 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:19,837 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:19,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:19,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:19,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,837 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:19,837 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,837 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:19,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416639840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416639840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416639840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416639842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416639844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,858 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255e85d0e87a2f4fabb7c3469b50c412a6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255e85d0e87a2f4fabb7c3469b50c412a6 2023-04-25 09:56:19,937 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:19,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:19,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:19,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:19,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:19,941 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:19,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:19,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:19,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:19,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,941 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:19,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:19,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:19,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:19,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:19,945 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416639945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416639945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:19,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:19,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416639946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416639946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d309418[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:19,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:19,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:19,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416639947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,949 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,950 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,950 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,950 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:19,952 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:19,955 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:19,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:19,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:19,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:19,955 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:19,956 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,956 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,957 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:19,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:19,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:19,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:19,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:19,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:19,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:19,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:19,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:19,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:19,961 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:19,961 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:19,961 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:19,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:20,061 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,064 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,067 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,067 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:20,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,071 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,071 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f911939[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,074 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,074 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,075 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,079 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,079 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,080 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,080 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,080 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,083 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,086 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,086 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,086 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,086 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416640148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416640148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416640148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416640149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416640151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,186 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,190 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,190 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,190 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,190 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,192 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,194 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,194 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,195 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:20,195 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,199 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e710742[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,215 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,222 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,222 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:20,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,264 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/cc66ba0e57c74b1a9edcb9aeb87d7662 2023-04-25 09:56:20,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/8266e7f1099447cb86f6357d1925bf68 2023-04-25 09:56:20,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/4b2933c90f394289bbc58b1686374ee9 2023-04-25 09:56:20,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/cc66ba0e57c74b1a9edcb9aeb87d7662 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/cc66ba0e57c74b1a9edcb9aeb87d7662 2023-04-25 09:56:20,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/cc66ba0e57c74b1a9edcb9aeb87d7662, entries=100, sequenceid=14, filesize=18.4 K 2023-04-25 09:56:20,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/8266e7f1099447cb86f6357d1925bf68 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/8266e7f1099447cb86f6357d1925bf68 2023-04-25 09:56:20,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/8266e7f1099447cb86f6357d1925bf68, entries=100, sequenceid=14, filesize=9.3 K 2023-04-25 09:56:20,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/4b2933c90f394289bbc58b1686374ee9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4b2933c90f394289bbc58b1686374ee9 2023-04-25 09:56:20,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4b2933c90f394289bbc58b1686374ee9, entries=100, sequenceid=14, filesize=9.3 K 2023-04-25 09:56:20,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for fb513dc5b3a62f2dba9b88a4e6de5394 in 498ms, sequenceid=14, compaction requested=false 2023-04-25 09:56:20,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:20,323 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,326 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,328 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,328 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,329 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,329 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,329 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-25 09:56:20,330 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:20,330 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:20,330 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:20,330 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:20,330 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:20,330 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:20,348 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042535c7b905e59641cba4040d5d0a6ef871 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042535c7b905e59641cba4040d5d0a6ef871 2023-04-25 09:56:20,358 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/8d35ef76f9384648b35497c4c33744ea 2023-04-25 09:56:20,372 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/2def4ad8c24b455f86bbe14a0f425af3 2023-04-25 09:56:20,386 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/5c5ce91e76df4d4fae119d550adb50fc 2023-04-25 09:56:20,389 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/8d35ef76f9384648b35497c4c33744ea as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d35ef76f9384648b35497c4c33744ea 2023-04-25 09:56:20,392 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d35ef76f9384648b35497c4c33744ea, entries=150, sequenceid=50, filesize=25.1 K 2023-04-25 09:56:20,393 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/2def4ad8c24b455f86bbe14a0f425af3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2def4ad8c24b455f86bbe14a0f425af3 2023-04-25 09:56:20,396 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2def4ad8c24b455f86bbe14a0f425af3, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 09:56:20,397 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/5c5ce91e76df4d4fae119d550adb50fc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5c5ce91e76df4d4fae119d550adb50fc 2023-04-25 09:56:20,400 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5c5ce91e76df4d4fae119d550adb50fc, entries=150, sequenceid=50, filesize=11.6 K 2023-04-25 09:56:20,401 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 72ms, sequenceid=50, compaction requested=false 2023-04-25 09:56:20,401 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:20,401 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,401 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,402 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,403 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,403 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ceb166c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,405 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,408 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,408 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,408 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,409 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,409 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,412 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,412 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,412 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,415 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,416 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,418 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,419 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,419 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,419 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:20,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:20,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:20,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:20,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:20,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:20,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:20,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:20,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:20,512 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304252f906ac1af484516a0b3922165504816 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252f906ac1af484516a0b3922165504816 2023-04-25 09:56:20,519 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416580505, max timestamp=1682416580519Num uniques 859; 2023-04-25 09:56:20,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:56:20,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682416580505, max timestamp=1682416580519Num uniques 859; 2023-04-25 09:56:20,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-25 09:56:20,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416580517, max timestamp=1682416580519Num uniques 859; 2023-04-25 09:56:20,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:20,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,531 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,532 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,534 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,534 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,535 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,535 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,540 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,540 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:20,541 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,541 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416640536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,542 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416640538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,542 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416640539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,543 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7502d3bd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,544 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,544 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,545 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,546 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416640541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416640541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,548 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,550 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,550 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,551 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,553 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,556 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,558 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,559 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,559 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:20,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:20,559 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416640642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416640643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416640644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416640647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416640647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,659 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,662 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,662 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,663 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,664 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,664 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:20,666 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,666 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,668 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,668 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62cd54f5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,670 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,670 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,675 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,675 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,675 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,676 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,679 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,679 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,679 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,680 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,681 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:20,686 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,686 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:20,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,786 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,790 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,790 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,790 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,790 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,790 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,790 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,791 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,792 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,793 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,793 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,793 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:20,793 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,793 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,794 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,800 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,801 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442014f9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,804 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,807 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,810 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,810 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,810 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,812 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,814 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,814 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,829 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,829 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:20,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,829 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416640845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416640846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416640848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416640848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:20,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416640850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,926 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/c23bd2f6f7884b868189ca5c6696ffd3 2023-04-25 09:56:20,929 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:20,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:20,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:20,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:20,933 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:20,933 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,933 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:20,933 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:20,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,934 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,935 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,936 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/10bbf78705d248a48f6c003670e82edc 2023-04-25 09:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:20,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:20,938 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,938 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:20,938 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:20,938 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:20,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102c3267[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:20,946 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:20,946 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,947 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:20,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:20,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:20,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,949 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:20,949 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:20,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:20,952 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:20,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:20,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:20,952 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:20,952 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:20,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:20,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:20,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:20,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:20,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:20,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:20,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:20,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:20,961 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:20,961 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,061 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,065 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,065 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,066 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,066 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,067 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,067 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:21,068 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,069 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,070 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,070 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,070 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,070 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73d094e4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,078 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,079 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,083 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:21,087 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,087 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416641149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416641150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416641152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416641152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416641153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,187 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,191 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,192 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,193 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,195 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,195 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,195 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:21,195 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,195 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,198 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b320c97[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,205 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,205 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,208 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,214 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,214 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,314 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,318 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,319 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,319 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,321 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,321 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,321 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,322 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:21,322 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,323 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,323 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff148d4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,325 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,327 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,331 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-25 09:56:21,334 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,334 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,335 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,338 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,338 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,338 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,338 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,338 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,342 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,343 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,343 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,349 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,349 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:21,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/0ee8bf3e86cd4ac58e42d65d484dbf0d 2023-04-25 09:56:21,354 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/c23bd2f6f7884b868189ca5c6696ffd3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c23bd2f6f7884b868189ca5c6696ffd3 2023-04-25 09:56:21,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c23bd2f6f7884b868189ca5c6696ffd3, entries=200, sequenceid=60, filesize=31.9 K 2023-04-25 09:56:21,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/10bbf78705d248a48f6c003670e82edc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/10bbf78705d248a48f6c003670e82edc 2023-04-25 09:56:21,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/10bbf78705d248a48f6c003670e82edc, entries=100, sequenceid=60, filesize=9.3 K 2023-04-25 09:56:21,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/0ee8bf3e86cd4ac58e42d65d484dbf0d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/0ee8bf3e86cd4ac58e42d65d484dbf0d 2023-04-25 09:56:21,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/0ee8bf3e86cd4ac58e42d65d484dbf0d, entries=100, sequenceid=60, filesize=9.3 K 2023-04-25 09:56:21,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=145.36 KB/148850 for fb513dc5b3a62f2dba9b88a4e6de5394 in 874ms, sequenceid=60, compaction requested=true 2023-04-25 09:56:21,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:21,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:21,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:21,369 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:21,369 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:21,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:21,370 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77226 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:21,370 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:21,370 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 30970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:21,370 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,370 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:21,370 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/cc66ba0e57c74b1a9edcb9aeb87d7662, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d35ef76f9384648b35497c4c33744ea, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c23bd2f6f7884b868189ca5c6696ffd3] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=75.4 K 2023-04-25 09:56:21,370 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,370 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/8266e7f1099447cb86f6357d1925bf68, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2def4ad8c24b455f86bbe14a0f425af3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/10bbf78705d248a48f6c003670e82edc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=30.2 K 2023-04-25 09:56:21,371 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting cc66ba0e57c74b1a9edcb9aeb87d7662, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682416579794 2023-04-25 09:56:21,371 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 8266e7f1099447cb86f6357d1925bf68, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682416579794 2023-04-25 09:56:21,371 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8d35ef76f9384648b35497c4c33744ea, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682416579834 2023-04-25 09:56:21,371 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2def4ad8c24b455f86bbe14a0f425af3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682416579834 2023-04-25 09:56:21,371 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c23bd2f6f7884b868189ca5c6696ffd3, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682416580474 2023-04-25 09:56:21,371 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 10bbf78705d248a48f6c003670e82edc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682416580493 2023-04-25 09:56:21,381 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#562 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:21,393 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#561 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:21,402 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/48ec436bdfb4499eaf943afbdb770930 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/48ec436bdfb4499eaf943afbdb770930 2023-04-25 09:56:21,406 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into 48ec436bdfb4499eaf943afbdb770930(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:21,406 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:21,406 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=13, startTime=1682416581369; duration=0sec 2023-04-25 09:56:21,406 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:21,406 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:21,407 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 30970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:21,407 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:21,408 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,408 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4b2933c90f394289bbc58b1686374ee9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5c5ce91e76df4d4fae119d550adb50fc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/0ee8bf3e86cd4ac58e42d65d484dbf0d] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=30.2 K 2023-04-25 09:56:21,408 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4b2933c90f394289bbc58b1686374ee9, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682416579794 2023-04-25 09:56:21,408 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5c5ce91e76df4d4fae119d550adb50fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682416579834 2023-04-25 09:56:21,409 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0ee8bf3e86cd4ac58e42d65d484dbf0d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682416580493 2023-04-25 09:56:21,421 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#563 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:21,447 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/f745bfa0182b46609026b06117303013 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f745bfa0182b46609026b06117303013 2023-04-25 09:56:21,449 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,451 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/f79a2495edc246afbdbd58cbba30dc25 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f79a2495edc246afbdbd58cbba30dc25 2023-04-25 09:56:21,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,453 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into f745bfa0182b46609026b06117303013(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:21,453 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:21,453 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=13, startTime=1682416581369; duration=0sec 2023-04-25 09:56:21,453 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:21,456 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into f79a2495edc246afbdbd58cbba30dc25(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:21,456 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:21,456 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=13, startTime=1682416581369; duration=0sec 2023-04-25 09:56:21,456 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,460 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,460 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,462 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,464 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,464 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,464 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,464 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=145.36 KB heapSize=405.69 KB 2023-04-25 09:56:21,464 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:21,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:21,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:21,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:21,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:21,465 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:21,498 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042590751f080f174b429f2e216309ecda04 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042590751f080f174b429f2e216309ecda04 2023-04-25 09:56:21,506 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/9ac3cac0113541b382a0877736bde983 2023-04-25 09:56:21,518 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/6c684ea3c8484d9885d0ee9bf511d101 2023-04-25 09:56:21,530 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/4be2f078448f41e3a762b75466e835e5 2023-04-25 09:56:21,535 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/9ac3cac0113541b382a0877736bde983 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9ac3cac0113541b382a0877736bde983 2023-04-25 09:56:21,540 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9ac3cac0113541b382a0877736bde983, entries=150, sequenceid=99, filesize=25.1 K 2023-04-25 09:56:21,541 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/6c684ea3c8484d9885d0ee9bf511d101 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6c684ea3c8484d9885d0ee9bf511d101 2023-04-25 09:56:21,545 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6c684ea3c8484d9885d0ee9bf511d101, entries=150, sequenceid=99, filesize=11.6 K 2023-04-25 09:56:21,545 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/4be2f078448f41e3a762b75466e835e5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4be2f078448f41e3a762b75466e835e5 2023-04-25 09:56:21,549 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4be2f078448f41e3a762b75466e835e5, entries=150, sequenceid=99, filesize=11.6 K 2023-04-25 09:56:21,549 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~405.64 KB/415376, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 85ms, sequenceid=99, compaction requested=false 2023-04-25 09:56:21,549 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:21,550 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,552 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,552 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3fea47[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,555 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,568 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,569 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,570 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,571 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,573 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,573 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,574 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,575 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,575 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,575 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,576 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,583 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,584 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,584 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,584 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:21,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:56:21,684 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:21,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,689 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,690 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,694 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,694 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,694 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:21,695 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,695 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,698 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425e3092573f90b4bd6adcdadf183d784c4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e3092573f90b4bd6adcdadf183d784c4 2023-04-25 09:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39bf30d0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,701 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,701 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,702 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,702 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,705 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,705 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416641702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416641703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416641706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416641707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416641707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,711 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,712 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,712 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,712 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,712 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:21,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416641808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416641808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,812 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416641812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416641812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416641812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,816 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,817 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,819 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,821 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,821 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,821 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:21,821 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,823 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5282d5c4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,826 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,827 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,828 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,831 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,831 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,831 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,834 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,834 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,836 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,840 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:21,840 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,940 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:21,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:21,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:21,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:21,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:21,943 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:21,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,943 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:21,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:21,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,945 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,945 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:21,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:21,947 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,947 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:21,947 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:21,947 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:21,947 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:21,949 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,949 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@670080a8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:21,951 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:21,951 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,953 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,953 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:21,953 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:21,953 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:21,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,953 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:21,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,954 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:21,957 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:21,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:21,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:21,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:21,957 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:21,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,959 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,959 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:21,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:21,959 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,960 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:21,967 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:21,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:21,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:21,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:21,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:21,967 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:21,967 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:21,967 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416642010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416642011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416642014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416642015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416642015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,067 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:22,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:22,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:22,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:22,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:22,071 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:22,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:22,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:22,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,072 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,073 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:22,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:22,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:22,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:22,074 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:22,074 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:22,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@565c33dd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:22,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:22,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,079 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:22,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:22,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,081 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:22,081 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,081 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:22,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:22,084 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:22,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:22,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:22,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:22,084 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:22,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,085 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,086 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,087 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,089 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:22,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:22,089 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,090 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:22,090 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:22,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:22,106 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/9a89a8a622d4451497819bc8c614b886 2023-04-25 09:56:22,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/de0390119bee4e6db6916d09c6c6bbab 2023-04-25 09:56:22,190 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:22,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:22,193 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:22,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:22,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:22,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,195 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,195 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,195 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,195 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:22,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:22,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:22,197 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,197 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:22,197 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:22,197 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:22,197 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:22,199 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,199 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,200 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,200 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a5b076[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:22,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:22,201 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:22,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:22,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:22,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:22,203 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,204 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:22,204 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,204 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:22,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:22,206 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:22,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:22,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:22,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:22,206 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:22,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,208 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:22,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:22,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,213 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,214 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,214 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,214 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:22,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,314 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:22,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416642314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416642315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:22,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:22,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:22,318 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:22,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:22,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:22,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416642318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416642318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,320 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:22,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416642319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:22,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:22,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:22,322 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,322 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:22,322 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:22,322 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:22,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52cd7b93[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:22,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:22,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:22,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:22,342 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:22,345 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:22,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:22,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:22,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:22,345 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:22,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,351 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:22,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:22,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,352 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:22,352 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:22,452 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:22,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:22,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:22,456 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:22,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:22,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:22,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,457 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:22,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:22,459 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,459 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:22,459 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:22,459 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:22,466 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31aba4a2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:22,468 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:22,468 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,469 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:22,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:22,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:22,471 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:22,474 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:22,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:22,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:22,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:22,474 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:22,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:22,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:22,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:22,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:22,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:22,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:22,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,481 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:22,481 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:22,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:22,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:22,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:22,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/abead543676344ab956db99267ad3e4e 2023-04-25 09:56:22,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/9a89a8a622d4451497819bc8c614b886 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9a89a8a622d4451497819bc8c614b886 2023-04-25 09:56:22,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9a89a8a622d4451497819bc8c614b886, entries=200, sequenceid=112, filesize=31.9 K 2023-04-25 09:56:22,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/de0390119bee4e6db6916d09c6c6bbab as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/de0390119bee4e6db6916d09c6c6bbab 2023-04-25 09:56:22,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/de0390119bee4e6db6916d09c6c6bbab, entries=150, sequenceid=112, filesize=11.6 K 2023-04-25 09:56:22,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/abead543676344ab956db99267ad3e4e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/abead543676344ab956db99267ad3e4e 2023-04-25 09:56:22,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/abead543676344ab956db99267ad3e4e, entries=150, sequenceid=112, filesize=11.6 K 2023-04-25 09:56:22,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for fb513dc5b3a62f2dba9b88a4e6de5394 in 855ms, sequenceid=112, compaction requested=true 2023-04-25 09:56:22,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:22,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:22,540 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:22,540 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:22,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:22,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:22,540 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:22,541 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:22,541 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,541 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f745bfa0182b46609026b06117303013, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9ac3cac0113541b382a0877736bde983, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9a89a8a622d4451497819bc8c614b886] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=82.3 K 2023-04-25 09:56:22,541 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:22,541 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:22,541 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,541 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/48ec436bdfb4499eaf943afbdb770930, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6c684ea3c8484d9885d0ee9bf511d101, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/de0390119bee4e6db6916d09c6c6bbab] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=34.9 K 2023-04-25 09:56:22,542 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f745bfa0182b46609026b06117303013, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682416579834 2023-04-25 09:56:22,542 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 48ec436bdfb4499eaf943afbdb770930, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682416579834 2023-04-25 09:56:22,542 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9ac3cac0113541b382a0877736bde983, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682416580534 2023-04-25 09:56:22,542 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6c684ea3c8484d9885d0ee9bf511d101, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682416580534 2023-04-25 09:56:22,542 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 9a89a8a622d4451497819bc8c614b886, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416581683 2023-04-25 09:56:22,542 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting de0390119bee4e6db6916d09c6c6bbab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416581683 2023-04-25 09:56:22,549 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#570 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:22,557 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#571 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:22,564 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/2207490c04a842ca9c0c0da88077d64f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2207490c04a842ca9c0c0da88077d64f 2023-04-25 09:56:22,567 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into 2207490c04a842ca9c0c0da88077d64f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:22,568 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:22,568 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=13, startTime=1682416582540; duration=0sec 2023-04-25 09:56:22,568 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:22,568 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:22,568 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:22,569 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:22,569 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,569 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f79a2495edc246afbdbd58cbba30dc25, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4be2f078448f41e3a762b75466e835e5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/abead543676344ab956db99267ad3e4e] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=34.9 K 2023-04-25 09:56:22,570 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f79a2495edc246afbdbd58cbba30dc25, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1682416579834 2023-04-25 09:56:22,570 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 4be2f078448f41e3a762b75466e835e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1682416580534 2023-04-25 09:56:22,571 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting abead543676344ab956db99267ad3e4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416581683 2023-04-25 09:56:22,577 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#572 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:22,581 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:22,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:22,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:22,596 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/e028db4d775244bb832188681eea166b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e028db4d775244bb832188681eea166b 2023-04-25 09:56:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:22,600 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:22,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:22,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:22,601 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,601 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,602 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into e028db4d775244bb832188681eea166b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:22,602 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:22,602 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=13, startTime=1682416582540; duration=0sec 2023-04-25 09:56:22,602 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:22,602 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:22,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:22,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:22,606 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/f0d861aa241c436f9ce1b6d31cee52a5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f0d861aa241c436f9ce1b6d31cee52a5 2023-04-25 09:56:22,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:22,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:22,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:22,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:22,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:22,611 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into f0d861aa241c436f9ce1b6d31cee52a5(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:22,611 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:22,611 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=13, startTime=1682416582540; duration=0sec 2023-04-25 09:56:22,611 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:22,616 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 09:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:22,620 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:22,620 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:22,620 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:22,620 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:22,620 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:22,635 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304257305856d111c43cebd5fe140a40c0ba7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257305856d111c43cebd5fe140a40c0ba7 2023-04-25 09:56:22,641 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/eef0298bc39247cc83bccdfe6acf237b 2023-04-25 09:56:22,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:22,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:22,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416642826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416642826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416642829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416642829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416642829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416642931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416642931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416642931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416642932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:22,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416642932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,053 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/75695efcd2334ba78f121fd20b020514 2023-04-25 09:56:23,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416643133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416643134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416643135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416643135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416643135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416643436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416643438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416643438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416643439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416643440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,465 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/6a99ba85684345f98f3da1077ffa262b 2023-04-25 09:56:23,469 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/eef0298bc39247cc83bccdfe6acf237b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/eef0298bc39247cc83bccdfe6acf237b 2023-04-25 09:56:23,472 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/eef0298bc39247cc83bccdfe6acf237b, entries=150, sequenceid=135, filesize=25.3 K 2023-04-25 09:56:23,473 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/75695efcd2334ba78f121fd20b020514 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/75695efcd2334ba78f121fd20b020514 2023-04-25 09:56:23,476 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/75695efcd2334ba78f121fd20b020514, entries=150, sequenceid=135, filesize=11.8 K 2023-04-25 09:56:23,477 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/6a99ba85684345f98f3da1077ffa262b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/6a99ba85684345f98f3da1077ffa262b 2023-04-25 09:56:23,480 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/6a99ba85684345f98f3da1077ffa262b, entries=150, sequenceid=135, filesize=11.8 K 2023-04-25 09:56:23,481 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fb513dc5b3a62f2dba9b88a4e6de5394 in 865ms, sequenceid=135, compaction requested=false 2023-04-25 09:56:23,481 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:23,481 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:23,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6deab91[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:23,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:23,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,486 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:23,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:23,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,486 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:23,489 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:23,492 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:23,493 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:23,493 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:23,493 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:23,493 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:23,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:23,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:23,494 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:23,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:23,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:23,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,499 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:23,499 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:23,499 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:23,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:23,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:23,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:23,599 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:23,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:23,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:23,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:23,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:23,603 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:23,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:23,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:23,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:23,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:23,604 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:23,604 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:23,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:23,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:23,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:23,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:23,606 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:23,606 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:23,606 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:23,606 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:23,606 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:23,606 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:23,606 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:23,606 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 09:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:23,607 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:23,619 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425cef11b978dc14406878f85d37de9a63a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cef11b978dc14406878f85d37de9a63a 2023-04-25 09:56:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:23,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:23,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416643951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416643952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416643953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416643954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:23,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416643954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,025 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/a0ff9369e3754f449100957369b08199 2023-04-25 09:56:24,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416644056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416644056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416644056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416644058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416644058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416644259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416644260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416644260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416644260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416644260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,433 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/928d3e6ce1564e8c98b3244ba8539e8f 2023-04-25 09:56:24,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416644563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416644563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416644563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416644563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:24,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416644564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,842 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/91cdda41fa5f43b48a6746a5400a233c 2023-04-25 09:56:24,846 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/a0ff9369e3754f449100957369b08199 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a0ff9369e3754f449100957369b08199 2023-04-25 09:56:24,850 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a0ff9369e3754f449100957369b08199, entries=150, sequenceid=147, filesize=25.3 K 2023-04-25 09:56:24,851 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/928d3e6ce1564e8c98b3244ba8539e8f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/928d3e6ce1564e8c98b3244ba8539e8f 2023-04-25 09:56:24,854 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/928d3e6ce1564e8c98b3244ba8539e8f, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 09:56:24,855 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/91cdda41fa5f43b48a6746a5400a233c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/91cdda41fa5f43b48a6746a5400a233c 2023-04-25 09:56:24,858 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/91cdda41fa5f43b48a6746a5400a233c, entries=150, sequenceid=147, filesize=11.8 K 2023-04-25 09:56:24,859 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fb513dc5b3a62f2dba9b88a4e6de5394 in 1253ms, sequenceid=147, compaction requested=true 2023-04-25 09:56:24,859 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:24,859 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:24,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:24,861 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:24,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:24,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d438d3c[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:24,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:24,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,865 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,865 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:24,865 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:24,865 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:24,865 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,865 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:24,866 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:24,867 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:24,869 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:24,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:24,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:24,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:24,869 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:24,871 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,871 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:24,871 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:24,872 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,874 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:24,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,876 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:24,876 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:24,876 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:24,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:24,876 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:24,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:24,976 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:24,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:24,980 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,980 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:24,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,981 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:24,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:24,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:24,982 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:24,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:24,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:24,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:24,983 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:24,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:24,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:24,983 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-25 09:56:24,983 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:24,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:24,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:24,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:24,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:24,984 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:25,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:25,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:25,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416645074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416645075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416645075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416645075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416645077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416645178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416645179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416645179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416645179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416645181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416645382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416645382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416645382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416645383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416645386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,397 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425150928f866c24f1a96a76a38ae478981 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425150928f866c24f1a96a76a38ae478981 2023-04-25 09:56:25,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416645686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416645686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416645686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416645687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:25,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416645692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:25,802 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/17f0fa17bbfc4eadb81e355ed097048f 2023-04-25 09:56:26,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416646188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416646189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:26,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416646192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416646193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416646198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,211 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/838e555a13e8406a95b9cb2d21db5ddd 2023-04-25 09:56:26,249 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/af09e97564b14bb1aebe31bd6e8764ad 2023-04-25 09:56:26,253 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/17f0fa17bbfc4eadb81e355ed097048f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/17f0fa17bbfc4eadb81e355ed097048f 2023-04-25 09:56:26,256 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/17f0fa17bbfc4eadb81e355ed097048f, entries=150, sequenceid=167, filesize=25.3 K 2023-04-25 09:56:26,257 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/838e555a13e8406a95b9cb2d21db5ddd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/838e555a13e8406a95b9cb2d21db5ddd 2023-04-25 09:56:26,260 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/838e555a13e8406a95b9cb2d21db5ddd, entries=150, sequenceid=167, filesize=11.8 K 2023-04-25 09:56:26,261 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/af09e97564b14bb1aebe31bd6e8764ad as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/af09e97564b14bb1aebe31bd6e8764ad 2023-04-25 09:56:26,270 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/af09e97564b14bb1aebe31bd6e8764ad, entries=150, sequenceid=167, filesize=11.8 K 2023-04-25 09:56:26,271 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fb513dc5b3a62f2dba9b88a4e6de5394 in 1288ms, sequenceid=167, compaction requested=true 2023-04-25 09:56:26,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:26,271 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:26,271 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:26,273 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb68e4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:26,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:26,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,279 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:26,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:26,279 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,280 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:26,279 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:26,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:26,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,281 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:26,281 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:26,284 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:26,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:26,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:26,285 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:26,285 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:26,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,287 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,288 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,292 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,295 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:26,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:26,295 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,296 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:26,296 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,296 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,296 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:26,396 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:26,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:26,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:26,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:26,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:26,399 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:26,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:26,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:26,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,413 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,414 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,414 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,414 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:26,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:26,415 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:26,415 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,415 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:26,416 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 09:56:26,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:26,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:26,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:26,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:26,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:26,416 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:26,427 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425bde1f9954ac8416482406d8d07b8051d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bde1f9954ac8416482406d8d07b8051d 2023-04-25 09:56:26,432 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=179, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/c3cc01e44a8746be9de6380174394010 2023-04-25 09:56:26,441 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/44eb7a8175114623a4d38c418dbe6e36 2023-04-25 09:56:26,450 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/ddeba9b3dc8a418e80906db2afd7f3b9 2023-04-25 09:56:26,454 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/c3cc01e44a8746be9de6380174394010 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c3cc01e44a8746be9de6380174394010 2023-04-25 09:56:26,457 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c3cc01e44a8746be9de6380174394010, entries=150, sequenceid=179, filesize=25.3 K 2023-04-25 09:56:26,458 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/44eb7a8175114623a4d38c418dbe6e36 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/44eb7a8175114623a4d38c418dbe6e36 2023-04-25 09:56:26,461 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/44eb7a8175114623a4d38c418dbe6e36, entries=150, sequenceid=179, filesize=11.8 K 2023-04-25 09:56:26,462 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/ddeba9b3dc8a418e80906db2afd7f3b9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/ddeba9b3dc8a418e80906db2afd7f3b9 2023-04-25 09:56:26,466 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/ddeba9b3dc8a418e80906db2afd7f3b9, entries=150, sequenceid=179, filesize=11.8 K 2023-04-25 09:56:26,468 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 52ms, sequenceid=179, compaction requested=true 2023-04-25 09:56:26,468 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:26,468 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:26,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:26,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:26,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:26,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,470 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:26,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6889526f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:26,472 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:26,472 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,473 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:26,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:26,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:26,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:26,477 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:26,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:26,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:26,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:26,478 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:26,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,483 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:26,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:26,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:26,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,486 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,486 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:26,486 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,486 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,486 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,486 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:26,586 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:26,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:26,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:26,590 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:26,590 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:26,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,591 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,592 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:26,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:26,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:26,594 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,594 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:26,594 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:26,594 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:26,594 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:26,595 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,595 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,596 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,596 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf8eecc[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:26,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:26,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,599 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:26,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,600 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:26,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:26,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:26,603 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:26,604 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:26,604 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:26,604 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:26,604 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:26,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,606 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,607 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:26,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:26,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:26,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,609 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,610 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,610 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:26,610 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,610 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,610 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:26,710 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:26,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:26,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:26,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:26,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:26,714 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:26,714 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,714 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:26,714 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:26,715 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,715 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,717 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:26,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:26,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:26,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:26,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:26,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:26,720 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,720 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:26,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:26,720 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:26,720 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:26,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:26,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:26,721 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:26,722 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,722 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,722 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,723 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a73aae2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:26,724 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:26,724 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,725 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:26,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:26,730 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:26,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:26,733 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:26,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:26,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:26,733 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:26,733 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:26,735 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,735 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,735 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,735 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,737 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,738 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,739 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:26,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:26,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,739 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,739 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:26,739 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:26,839 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:26,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:26,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:26,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:26,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:26,846 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:26,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:26,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:26,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,848 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,848 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,848 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,848 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:26,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:26,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:26,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:26,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:26,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:26,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:26,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:26,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:26,854 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:26,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65805900[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:26,856 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:26,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:26,858 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:26,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,858 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:26,859 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,859 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:26,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:26,862 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:26,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:26,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:26,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:26,862 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:26,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,863 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,864 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,864 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,865 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:26,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:26,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,869 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:26,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,870 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:26,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,870 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:26,970 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:26,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:26,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:26,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:26,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:26,977 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,978 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:26,978 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,984 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,984 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:26,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:26,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:26,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:26,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:26,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:26,986 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:26,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:26,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473875f5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:26,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,991 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:26,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,991 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:26,994 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:26,994 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:26,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,994 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:26,996 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:26,996 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:26,996 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:26,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:26,998 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:26,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:26,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,000 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,003 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,004 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,004 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,004 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,104 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,113 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,118 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,118 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,119 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:27,119 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,124 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b6a7d50[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,126 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,126 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,127 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,127 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,127 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,128 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,131 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,131 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,131 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,131 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,131 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,133 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,134 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,145 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,145 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:27,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,145 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,145 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,146 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,146 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,146 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,146 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,146 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,146 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,146 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,146 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,246 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,250 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,250 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:27,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:27,254 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,254 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,254 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,254 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,254 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:27,254 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:27,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:27,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:27,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:27,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:27,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,256 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,256 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,256 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:27,257 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,258 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c69f065[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,260 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,260 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,262 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,263 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,263 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,264 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,267 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,267 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,267 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,269 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,269 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,269 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,271 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,272 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,275 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,275 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,276 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,276 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,280 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416647278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416647279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416647280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416587273, max timestamp=1682416587281Num uniques 859; 2023-04-25 09:56:27,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:27,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416587273, max timestamp=1682416587281Num uniques 859; 2023-04-25 09:56:27,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:27,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682416587273, max timestamp=1682416587281Num uniques 859; 2023-04-25 09:56:27,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-25 09:56:27,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416647311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416647312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,376 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,380 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,380 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,382 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,382 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,383 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,383 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,384 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,384 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,385 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:27,385 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,386 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,386 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416647384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416647384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416647384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc6f6b1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,390 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,391 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,392 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,393 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,395 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,395 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,395 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,396 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,396 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,398 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,398 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,398 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,399 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,400 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,402 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:27,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,402 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,402 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,402 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416647416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416647418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,502 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,506 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,506 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,506 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,506 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,506 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,507 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,508 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,508 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,508 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,508 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,510 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,511 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,511 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,512 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd5b7e1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,514 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,517 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,517 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,520 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,522 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,522 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,522 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,523 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,523 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,524 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,524 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,524 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,525 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,527 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,531 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,531 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,531 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,531 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416647588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416647588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416647591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416647619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416647624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,632 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,635 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,636 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,636 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,637 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,637 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,637 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:27,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,643 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,643 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,643 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:27,643 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,644 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,645 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@179dd1b6[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,648 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,648 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,649 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,649 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,652 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,655 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,655 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,655 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,657 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,657 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,659 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,661 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,662 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,662 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,661 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,662 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,669 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255d7f0f12e2784ab49dd4f9458cfea30b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255d7f0f12e2784ab49dd4f9458cfea30b 2023-04-25 09:56:27,674 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/49946d52b48b498f81390e14283f3410 2023-04-25 09:56:27,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/2fe9e2889c13467c9023bdfcb8c580b9 2023-04-25 09:56:27,762 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,765 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,765 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,766 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,767 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,767 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,767 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,767 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,769 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,769 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,769 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:27,769 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,769 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,776 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6484e1f8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,778 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,779 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,779 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,779 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,780 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,781 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,783 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,783 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,783 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,785 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,786 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,787 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,790 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,791 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,791 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,791 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,791 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,791 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:27,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,891 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:27,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:27,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:27,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:27,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:27,895 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:27,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:27,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:27,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,895 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416647893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416647893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,897 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,897 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:27,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:27,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:27,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:27,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:27,898 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:27,898 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,899 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:27,899 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:27,899 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:27,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:27,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416647896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:27,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:27,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:27,899 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:27,900 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@89bf0a7[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:27,902 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:27,902 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,903 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:27,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,904 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:27,909 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:27,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:27,912 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:27,913 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:27,913 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:27,913 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:27,913 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:27,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,914 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:27,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,915 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,917 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,919 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,919 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:27,919 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,919 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:27,919 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:27,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:27,920 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:27,920 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:27,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:27,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:27,921 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:27,921 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:27,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416647923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:27,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:27,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416647926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,021 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:28,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:28,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:28,024 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:28,024 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:28,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:28,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:28,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,025 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,034 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:28,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:28,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,035 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:28,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:28,036 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:28,036 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:28,036 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:28,036 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:28,038 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,038 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8334ea[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:28,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:28,040 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,041 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:28,041 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:28,041 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:28,041 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:28,042 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:28,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:28,042 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:28,045 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:28,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:28,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:28,045 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:28,046 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:28,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:28,048 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:28,050 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,051 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:28,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:28,053 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:28,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:28,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:28,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:28,053 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:28,054 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,054 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:28,054 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:28,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:28,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/199d0f6c6d0a42df824f6a456fd48b2b 2023-04-25 09:56:28,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/49946d52b48b498f81390e14283f3410 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49946d52b48b498f81390e14283f3410 2023-04-25 09:56:28,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49946d52b48b498f81390e14283f3410, entries=250, sequenceid=195, filesize=38.9 K 2023-04-25 09:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/2fe9e2889c13467c9023bdfcb8c580b9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2fe9e2889c13467c9023bdfcb8c580b9 2023-04-25 09:56:28,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2fe9e2889c13467c9023bdfcb8c580b9, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 09:56:28,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/199d0f6c6d0a42df824f6a456fd48b2b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/199d0f6c6d0a42df824f6a456fd48b2b 2023-04-25 09:56:28,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/199d0f6c6d0a42df824f6a456fd48b2b, entries=150, sequenceid=195, filesize=11.8 K 2023-04-25 09:56:28,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fb513dc5b3a62f2dba9b88a4e6de5394 in 857ms, sequenceid=195, compaction requested=true 2023-04-25 09:56:28,110 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:28,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:28,111 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:28,111 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:28,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:28,111 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:28,112 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 169416 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:28,112 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72272 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:28,112 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:28,112 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:28,112 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:28,112 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:28,113 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f0d861aa241c436f9ce1b6d31cee52a5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/eef0298bc39247cc83bccdfe6acf237b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a0ff9369e3754f449100957369b08199, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/17f0fa17bbfc4eadb81e355ed097048f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c3cc01e44a8746be9de6380174394010, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49946d52b48b498f81390e14283f3410] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=165.4 K 2023-04-25 09:56:28,113 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2207490c04a842ca9c0c0da88077d64f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/75695efcd2334ba78f121fd20b020514, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/928d3e6ce1564e8c98b3244ba8539e8f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/838e555a13e8406a95b9cb2d21db5ddd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/44eb7a8175114623a4d38c418dbe6e36, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2fe9e2889c13467c9023bdfcb8c580b9] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=70.6 K 2023-04-25 09:56:28,113 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2207490c04a842ca9c0c0da88077d64f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416581683 2023-04-25 09:56:28,113 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f0d861aa241c436f9ce1b6d31cee52a5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416581683 2023-04-25 09:56:28,114 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 75695efcd2334ba78f121fd20b020514, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682416581705 2023-04-25 09:56:28,114 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting eef0298bc39247cc83bccdfe6acf237b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682416581705 2023-04-25 09:56:28,114 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 928d3e6ce1564e8c98b3244ba8539e8f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682416582825 2023-04-25 09:56:28,114 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a0ff9369e3754f449100957369b08199, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682416582825 2023-04-25 09:56:28,114 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 838e555a13e8406a95b9cb2d21db5ddd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682416583951 2023-04-25 09:56:28,114 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 17f0fa17bbfc4eadb81e355ed097048f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682416583951 2023-04-25 09:56:28,115 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 44eb7a8175114623a4d38c418dbe6e36, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682416585073 2023-04-25 09:56:28,115 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting c3cc01e44a8746be9de6380174394010, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682416585073 2023-04-25 09:56:28,115 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2fe9e2889c13467c9023bdfcb8c580b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682416587251 2023-04-25 09:56:28,115 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 49946d52b48b498f81390e14283f3410, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682416587240 2023-04-25 09:56:28,125 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#588 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:28,129 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#589 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:28,136 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/0d89b868835744c9876993c5f0c2d3b4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0d89b868835744c9876993c5f0c2d3b4 2023-04-25 09:56:28,140 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into 0d89b868835744c9876993c5f0c2d3b4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:28,140 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:28,140 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=10, startTime=1682416588111; duration=0sec 2023-04-25 09:56:28,140 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:28,140 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-25 09:56:28,141 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72272 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-25 09:56:28,142 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:28,142 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:28,142 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e028db4d775244bb832188681eea166b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/6a99ba85684345f98f3da1077ffa262b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/91cdda41fa5f43b48a6746a5400a233c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/af09e97564b14bb1aebe31bd6e8764ad, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/ddeba9b3dc8a418e80906db2afd7f3b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/199d0f6c6d0a42df824f6a456fd48b2b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=70.6 K 2023-04-25 09:56:28,142 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e028db4d775244bb832188681eea166b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1682416581683 2023-04-25 09:56:28,142 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6a99ba85684345f98f3da1077ffa262b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682416581705 2023-04-25 09:56:28,143 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 91cdda41fa5f43b48a6746a5400a233c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682416582825 2023-04-25 09:56:28,143 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting af09e97564b14bb1aebe31bd6e8764ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682416583951 2023-04-25 09:56:28,143 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ddeba9b3dc8a418e80906db2afd7f3b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682416585073 2023-04-25 09:56:28,143 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 199d0f6c6d0a42df824f6a456fd48b2b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682416587251 2023-04-25 09:56:28,152 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#590 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:28,154 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:28,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:28,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:28,158 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:28,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:28,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:28,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:28,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,158 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,159 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/9516e23f55d5469e94c7641eb03fe9b3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/9516e23f55d5469e94c7641eb03fe9b3 2023-04-25 09:56:28,160 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:28,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:28,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,160 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:28,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:28,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:28,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:28,166 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into 9516e23f55d5469e94c7641eb03fe9b3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:28,166 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:28,166 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=10, startTime=1682416588111; duration=0sec 2023-04-25 09:56:28,166 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:28,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:28,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:28,168 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:28,168 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-25 09:56:28,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:28,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:28,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:28,169 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:28,169 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:28,169 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:28,181 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425c52854df27a447eb8a9a80bb767f8336 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c52854df27a447eb8a9a80bb767f8336 2023-04-25 09:56:28,185 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/417cd5404540469591f690d4e071da97 2023-04-25 09:56:28,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:28,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:28,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416648408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416648409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416648412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416648425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416648431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416648513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416648514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416648516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,601 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/6f2a1f16ac5c4e33895addfe4dce39bd 2023-04-25 09:56:28,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416648718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416648719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:28,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416648719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:28,951 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/947d80a73db3471d8d15d05c9522a8ae as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/947d80a73db3471d8d15d05c9522a8ae 2023-04-25 09:56:28,956 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into 947d80a73db3471d8d15d05c9522a8ae(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:28,956 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:28,956 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=10, startTime=1682416588111; duration=0sec 2023-04-25 09:56:28,956 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:29,010 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/2ebae39ad66f4bda92c51ba516010377 2023-04-25 09:56:29,014 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/417cd5404540469591f690d4e071da97 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/417cd5404540469591f690d4e071da97 2023-04-25 09:56:29,018 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/417cd5404540469591f690d4e071da97, entries=150, sequenceid=231, filesize=25.3 K 2023-04-25 09:56:29,019 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/6f2a1f16ac5c4e33895addfe4dce39bd as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6f2a1f16ac5c4e33895addfe4dce39bd 2023-04-25 09:56:29,022 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6f2a1f16ac5c4e33895addfe4dce39bd, entries=150, sequenceid=231, filesize=11.8 K 2023-04-25 09:56:29,023 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/2ebae39ad66f4bda92c51ba516010377 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/2ebae39ad66f4bda92c51ba516010377 2023-04-25 09:56:29,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416649022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416649023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416649024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,027 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/2ebae39ad66f4bda92c51ba516010377, entries=150, sequenceid=231, filesize=11.8 K 2023-04-25 09:56:29,027 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for fb513dc5b3a62f2dba9b88a4e6de5394 in 859ms, sequenceid=231, compaction requested=false 2023-04-25 09:56:29,027 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:29,027 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,029 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a5a6d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,032 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,032 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,034 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,034 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,035 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,038 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,038 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,038 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,038 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,038 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,040 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,041 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,042 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,044 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,044 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,045 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,045 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,045 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,145 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,148 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,148 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,149 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,151 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,151 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,152 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,153 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:29,153 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:29,176 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304251d8099cad97c4b60a5bb2713ac400a92 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251d8099cad97c4b60a5bb2713ac400a92 2023-04-25 09:56:29,185 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/8d9a9be6a24a4c0393705ab5be890897 2023-04-25 09:56:29,196 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/3067cce0113243d9b1f79859d5d39066 2023-04-25 09:56:29,207 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/5e9fbf1c80324b67bbfe66f4a742b70f 2023-04-25 09:56:29,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/8d9a9be6a24a4c0393705ab5be890897 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d9a9be6a24a4c0393705ab5be890897 2023-04-25 09:56:29,214 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d9a9be6a24a4c0393705ab5be890897, entries=150, sequenceid=243, filesize=25.3 K 2023-04-25 09:56:29,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/3067cce0113243d9b1f79859d5d39066 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/3067cce0113243d9b1f79859d5d39066 2023-04-25 09:56:29,218 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/3067cce0113243d9b1f79859d5d39066, entries=150, sequenceid=243, filesize=11.8 K 2023-04-25 09:56:29,218 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/5e9fbf1c80324b67bbfe66f4a742b70f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5e9fbf1c80324b67bbfe66f4a742b70f 2023-04-25 09:56:29,222 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5e9fbf1c80324b67bbfe66f4a742b70f, entries=150, sequenceid=243, filesize=11.8 K 2023-04-25 09:56:29,223 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 70ms, sequenceid=243, compaction requested=true 2023-04-25 09:56:29,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:29,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,224 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4022f0d9[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,229 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,229 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,232 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,232 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,234 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,236 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,238 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:29,239 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,239 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,239 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,239 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,339 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,343 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,344 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,344 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:29,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,349 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176b3eb2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,367 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,367 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,367 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,369 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,372 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,372 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,374 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,375 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,378 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,378 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,378 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,378 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:29,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:29,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:29,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:29,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:29,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:29,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:29,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:29,478 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,482 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,486 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,487 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,487 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,488 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:29,488 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,488 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,489 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416589478, max timestamp=1682416589487Num uniques 859; 2023-04-25 09:56:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416589478, max timestamp=1682416589487Num uniques 859; 2023-04-25 09:56:29,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:29,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416589478, max timestamp=1682416589487Num uniques 859; 2023-04-25 09:56:29,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ac6cd1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,493 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,493 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,494 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,494 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,494 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,494 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,495 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,498 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,498 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,498 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,501 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,503 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,505 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:29,505 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,505 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,505 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416649519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48616 deadline: 1682416649521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416649529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416649529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416649529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,605 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,609 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,610 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:29,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2544e91b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,618 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,618 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,619 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,619 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,619 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,620 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,623 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,623 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,623 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,623 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,623 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416649625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416649625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,633 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,633 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,633 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,634 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,635 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,637 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,638 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,638 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:29,638 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,638 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,738 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,742 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,742 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,744 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,744 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,745 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:29,746 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,748 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,748 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72377152[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,750 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,750 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,751 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,751 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,755 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,755 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,756 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,758 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,758 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,758 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,758 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,758 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,759 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,762 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,764 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,764 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,764 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,765 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416649827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:29,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48692 deadline: 1682416649829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,865 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,869 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,869 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,870 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,870 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,871 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,871 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,872 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,872 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,872 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:29,873 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,873 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,875 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,875 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c675f08[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,877 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,878 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:29,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,878 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:29,879 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,880 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,881 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042574e14a453ad24c9eb9e298a98297de0f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042574e14a453ad24c9eb9e298a98297de0f 2023-04-25 09:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:29,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:29,882 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:29,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:29,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:29,883 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:29,883 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:29,884 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,884 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,884 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,885 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,886 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:29,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:29,888 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:29,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:29,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,888 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,889 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,889 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:29,889 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:29,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:29,894 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/579a016c964c4a27a4e14fcee23f2473 2023-04-25 09:56:29,989 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:29,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:29,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:29,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:29,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:29,992 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:29,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:29,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:29,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,994 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,994 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:29,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:29,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,995 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:29,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:29,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:29,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:29,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:29,996 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:29,996 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:29,997 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,997 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:29,998 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bba76b1[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:29,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:29,999 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,000 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:30,000 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:30,000 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,000 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:30,001 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:30,003 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:30,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:30,003 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:30,004 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:30,004 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:30,006 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,006 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,006 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,006 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,007 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:30,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:30,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:30,009 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,010 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,010 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:30,009 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,010 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,110 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:30,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:30,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:30,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:30,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:30,113 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:30,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:30,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:30,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,114 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:30,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:30,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:30,118 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,118 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:30,118 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:30,118 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,120 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b44eec[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:30,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:30,123 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,128 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,128 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:30,129 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:30,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,129 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:30,130 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,130 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:30,132 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:30,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:30,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:30,132 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:30,132 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:30,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416650129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,133 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,134 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,135 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416650133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,141 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,141 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:30,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:30,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,142 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:30,142 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:30,242 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:30,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:30,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:30,245 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,245 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,246 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,247 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:30,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:30,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:30,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:30,248 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:30,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:30,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:30,249 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:30,250 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,250 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60401769[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:30,252 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:30,252 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,253 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,253 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,253 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:30,253 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:30,256 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,256 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:30,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:30,259 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:30,259 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:30,259 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:30,259 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:30,259 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:30,260 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,261 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,261 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,261 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,263 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,265 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:30,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:30,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,265 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,266 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,266 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,266 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:30,266 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,266 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:30,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/299845aabe7e4304969bb5fbdeedad30 2023-04-25 09:56:30,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/a191433db0984f0fa7696ef3e31f22b1 2023-04-25 09:56:30,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/579a016c964c4a27a4e14fcee23f2473 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/579a016c964c4a27a4e14fcee23f2473 2023-04-25 09:56:30,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/579a016c964c4a27a4e14fcee23f2473, entries=200, sequenceid=254, filesize=32.1 K 2023-04-25 09:56:30,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/299845aabe7e4304969bb5fbdeedad30 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/299845aabe7e4304969bb5fbdeedad30 2023-04-25 09:56:30,328 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/299845aabe7e4304969bb5fbdeedad30, entries=150, sequenceid=254, filesize=11.8 K 2023-04-25 09:56:30,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/a191433db0984f0fa7696ef3e31f22b1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a191433db0984f0fa7696ef3e31f22b1 2023-04-25 09:56:30,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a191433db0984f0fa7696ef3e31f22b1, entries=150, sequenceid=254, filesize=11.8 K 2023-04-25 09:56:30,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fb513dc5b3a62f2dba9b88a4e6de5394 in 880ms, sequenceid=254, compaction requested=true 2023-04-25 09:56:30,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:30,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:30,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:30,332 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:30,332 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:30,332 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:30,333 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:30,334 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:30,334 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,334 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/947d80a73db3471d8d15d05c9522a8ae, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/417cd5404540469591f690d4e071da97, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d9a9be6a24a4c0393705ab5be890897, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/579a016c964c4a27a4e14fcee23f2473] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=108.4 K 2023-04-25 09:56:30,334 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:30,334 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:30,334 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,334 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0d89b868835744c9876993c5f0c2d3b4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6f2a1f16ac5c4e33895addfe4dce39bd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/3067cce0113243d9b1f79859d5d39066, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/299845aabe7e4304969bb5fbdeedad30] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=47.4 K 2023-04-25 09:56:30,335 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 947d80a73db3471d8d15d05c9522a8ae, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682416587251 2023-04-25 09:56:30,335 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0d89b868835744c9876993c5f0c2d3b4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682416587251 2023-04-25 09:56:30,335 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 417cd5404540469591f690d4e071da97, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682416587299 2023-04-25 09:56:30,335 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 6f2a1f16ac5c4e33895addfe4dce39bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682416587299 2023-04-25 09:56:30,335 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 8d9a9be6a24a4c0393705ab5be890897, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682416588402 2023-04-25 09:56:30,335 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3067cce0113243d9b1f79859d5d39066, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682416588402 2023-04-25 09:56:30,336 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 579a016c964c4a27a4e14fcee23f2473, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682416589445 2023-04-25 09:56:30,336 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 299845aabe7e4304969bb5fbdeedad30, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682416589449 2023-04-25 09:56:30,345 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#600 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:30,354 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#601 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:30,358 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/e7bd06f7a8f8458ca9dd5c78dc55fa50 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/e7bd06f7a8f8458ca9dd5c78dc55fa50 2023-04-25 09:56:30,364 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into e7bd06f7a8f8458ca9dd5c78dc55fa50(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:30,364 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:30,364 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=12, startTime=1682416590332; duration=0sec 2023-04-25 09:56:30,364 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:30,364 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-25 09:56:30,365 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-25 09:56:30,365 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:30,366 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,366 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/9516e23f55d5469e94c7641eb03fe9b3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/2ebae39ad66f4bda92c51ba516010377, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5e9fbf1c80324b67bbfe66f4a742b70f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a191433db0984f0fa7696ef3e31f22b1] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=47.4 K 2023-04-25 09:56:30,366 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9516e23f55d5469e94c7641eb03fe9b3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682416587251 2023-04-25 09:56:30,366 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:30,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:30,367 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 2ebae39ad66f4bda92c51ba516010377, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682416587299 2023-04-25 09:56:30,367 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 5e9fbf1c80324b67bbfe66f4a742b70f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682416588402 2023-04-25 09:56:30,367 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a191433db0984f0fa7696ef3e31f22b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682416589449 2023-04-25 09:56:30,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:30,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:30,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:30,369 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:30,369 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,369 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:30,369 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:30,370 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,370 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,371 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,371 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,371 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,371 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:30,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:30,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:30,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:30,373 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-25 09:56:30,373 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:30,374 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:30,375 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:30,388 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304253c769a3c823f4972bc5f83361cc032e0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253c769a3c823f4972bc5f83361cc032e0 2023-04-25 09:56:30,393 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=292, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/df7969ab47b34195bed32ac6ac2c1d9b 2023-04-25 09:56:30,402 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/0cfe0fc552d548288c56e67c556bfd18 2023-04-25 09:56:30,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:30,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:30,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416650545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416650546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416650548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416650634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416650640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416650649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416650650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416650650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,788 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/b882a59003014c10b541f821ff369253 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b882a59003014c10b541f821ff369253 2023-04-25 09:56:30,791 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into b882a59003014c10b541f821ff369253(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:30,791 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:30,791 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=12, startTime=1682416590332; duration=0sec 2023-04-25 09:56:30,791 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:30,814 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/afaca8477b87428e92b9a224c8e8b358 2023-04-25 09:56:30,819 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/df7969ab47b34195bed32ac6ac2c1d9b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/df7969ab47b34195bed32ac6ac2c1d9b 2023-04-25 09:56:30,823 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/df7969ab47b34195bed32ac6ac2c1d9b, entries=150, sequenceid=292, filesize=25.4 K 2023-04-25 09:56:30,824 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/0cfe0fc552d548288c56e67c556bfd18 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0cfe0fc552d548288c56e67c556bfd18 2023-04-25 09:56:30,827 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0cfe0fc552d548288c56e67c556bfd18, entries=150, sequenceid=292, filesize=11.9 K 2023-04-25 09:56:30,827 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/afaca8477b87428e92b9a224c8e8b358 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/afaca8477b87428e92b9a224c8e8b358 2023-04-25 09:56:30,833 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/afaca8477b87428e92b9a224c8e8b358, entries=150, sequenceid=292, filesize=11.9 K 2023-04-25 09:56:30,834 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for fb513dc5b3a62f2dba9b88a4e6de5394 in 461ms, sequenceid=292, compaction requested=false 2023-04-25 09:56:30,834 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:30,834 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,836 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,837 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:30,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d99b76[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:30,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:30,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,839 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:30,839 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:30,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:30,841 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:30,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:30,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:30,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:30,844 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:30,844 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:30,845 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,845 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,845 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,850 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:30,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:30,851 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:30,851 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:30,851 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:30,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:30,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-25 09:56:30,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:30,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:30,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:30,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:30,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:30,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:30,872 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425091e3f85b2614dc68888a182b890d7c9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425091e3f85b2614dc68888a182b890d7c9 2023-04-25 09:56:30,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416650886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416650887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416650889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,951 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:30,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:30,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:30,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:30,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:30,954 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:30,954 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:30,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:30,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,955 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,956 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,956 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,957 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:30,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:30,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:30,958 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,958 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:30,958 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:30,958 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:30,958 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:30,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:30,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:30,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e2e66e2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:30,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:30,962 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,963 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:30,963 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,963 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:30,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:30,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:30,968 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:30,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:30,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:30,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:30,968 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:30,969 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:30,969 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,969 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:30,970 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:30,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:30,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:30,977 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:30,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:30,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:30,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:30,977 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:30,977 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:30,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416650991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416650991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:30,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416650992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,078 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,081 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,082 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,083 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,084 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,085 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,085 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,085 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,086 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,086 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,086 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,086 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,087 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@156698e5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,090 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,090 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,093 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,095 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,095 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,103 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,103 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,103 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,103 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,103 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,176 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/a68bc2680f5f46f1b20c03524cab53b1 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a68bc2680f5f46f1b20c03524cab53b1 2023-04-25 09:56:31,179 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into a68bc2680f5f46f1b20c03524cab53b1(size=25.8 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:31,180 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:31,180 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=12, startTime=1682416590332; duration=0sec 2023-04-25 09:56:31,180 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:31,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416651195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416651196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416651196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,203 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,206 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,207 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,208 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,208 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,209 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,210 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,210 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,211 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,212 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,213 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75fd3e7f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,214 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,214 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,217 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,217 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,219 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,222 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,222 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,224 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,229 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,229 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,289 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/519a6b4d3b904f9e852c776c86c767ed 2023-04-25 09:56:31,329 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,332 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,332 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,332 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,332 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,333 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,335 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,337 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,337 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,337 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,337 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,338 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f55199d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,341 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,341 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,342 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,343 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,345 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,345 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,348 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,350 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,353 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,353 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,353 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,453 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,457 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,461 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,461 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,463 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,463 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,465 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,465 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a16de3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,467 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,469 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,469 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,470 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,470 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,473 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,473 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,473 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,474 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,480 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,480 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416651498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416651499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416651501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,580 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,584 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,584 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,585 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,586 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,586 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,586 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,587 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,588 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,588 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,588 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,588 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,589 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,590 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13217ae2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,592 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,593 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,593 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,593 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,596 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,599 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,599 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,599 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,599 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,599 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,600 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,601 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,602 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,606 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,606 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,606 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,606 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416651645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:31,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416651645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/90a9250f2bbd4337a968e81456954e90 2023-04-25 09:56:31,706 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,709 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,709 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,710 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,711 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,711 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,712 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,712 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,713 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,715 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,715 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fb3b03f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,717 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,718 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,718 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,719 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,722 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,722 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,722 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,724 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,726 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,734 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,735 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,735 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,735 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,734 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,735 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,735 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,835 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,838 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,838 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,839 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,840 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,840 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,842 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,842 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,842 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,842 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,842 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,844 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,844 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785dc9a5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,846 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,850 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,851 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,852 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,855 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,855 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,856 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,857 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,859 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,861 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:31,861 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,861 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,862 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,862 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,862 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,862 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,962 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:31,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:31,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:31,965 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:31,965 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,965 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:31,965 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:31,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,968 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,968 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,969 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,969 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:31,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:31,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:31,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:31,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:31,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:31,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:31,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:31,970 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:31,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:31,970 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:31,971 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:31,972 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,972 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:31,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41aa6d38[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:31,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:31,975 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,975 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:31,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,976 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:31,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:31,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:31,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:31,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:31,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:31,981 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:31,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:31,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:31,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:31,981 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:31,983 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,984 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,984 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:31,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,984 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:31,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:31,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:31,987 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:31,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:31,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:31,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:31,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:31,988 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:31,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:31,988 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:31,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:32,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:32,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48606 deadline: 1682416652002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:32,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48690 deadline: 1682416652004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:32,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416652006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,088 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:32,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:32,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:32,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:32,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:32,093 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:32,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:32,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:32,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:32,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,093 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,094 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:32,094 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:32,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,095 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:32,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:32,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:32,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:32,096 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:32,096 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:32,097 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:32,097 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:32,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:32,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:32,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:32,097 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:32,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,098 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:32,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:32,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:32,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:32,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5944e985[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:32,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:32,100 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,102 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:32,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:32,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:32,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,103 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:32,104 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:32,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:32,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:32,104 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/bf133d1362f546cea4fa8fd923af39da 2023-04-25 09:56:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:32,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:32,107 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:32,107 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:32,107 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:32,107 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:32,107 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:32,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:32,109 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:32,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,110 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,110 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/519a6b4d3b904f9e852c776c86c767ed as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/519a6b4d3b904f9e852c776c86c767ed 2023-04-25 09:56:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:32,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/519a6b4d3b904f9e852c776c86c767ed, entries=250, sequenceid=306, filesize=39.2 K 2023-04-25 09:56:32,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:32,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:32,114 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:32,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:32,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:32,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/90a9250f2bbd4337a968e81456954e90 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/90a9250f2bbd4337a968e81456954e90 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:32,115 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:32,115 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:32,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:32,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:32,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/90a9250f2bbd4337a968e81456954e90, entries=150, sequenceid=306, filesize=11.9 K 2023-04-25 09:56:32,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/bf133d1362f546cea4fa8fd923af39da as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/bf133d1362f546cea4fa8fd923af39da 2023-04-25 09:56:32,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/bf133d1362f546cea4fa8fd923af39da, entries=150, sequenceid=306, filesize=11.9 K 2023-04-25 09:56:32,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for fb513dc5b3a62f2dba9b88a4e6de5394 in 1267ms, sequenceid=306, compaction requested=true 2023-04-25 09:56:32,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:32,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:32,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:32,126 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:32,126 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:32,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:32,127 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92618 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:32,127 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:32,127 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:32,127 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a68bc2680f5f46f1b20c03524cab53b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/df7969ab47b34195bed32ac6ac2c1d9b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/519a6b4d3b904f9e852c776c86c767ed] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=90.4 K 2023-04-25 09:56:32,127 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:32,128 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:32,128 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:32,128 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/e7bd06f7a8f8458ca9dd5c78dc55fa50, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0cfe0fc552d548288c56e67c556bfd18, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/90a9250f2bbd4337a968e81456954e90] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.1 K 2023-04-25 09:56:32,128 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting a68bc2680f5f46f1b20c03524cab53b1, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682416589449 2023-04-25 09:56:32,128 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting e7bd06f7a8f8458ca9dd5c78dc55fa50, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682416589449 2023-04-25 09:56:32,128 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting df7969ab47b34195bed32ac6ac2c1d9b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682416589517 2023-04-25 09:56:32,128 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 0cfe0fc552d548288c56e67c556bfd18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682416589517 2023-04-25 09:56:32,129 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 519a6b4d3b904f9e852c776c86c767ed, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682416590545 2023-04-25 09:56:32,129 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 90a9250f2bbd4337a968e81456954e90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682416590858 2023-04-25 09:56:32,135 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#610 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:32,137 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#609 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:32,160 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/a1d15e9e437b4fab86cfb4b17e2c8334 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a1d15e9e437b4fab86cfb4b17e2c8334 2023-04-25 09:56:32,166 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into a1d15e9e437b4fab86cfb4b17e2c8334(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:32,166 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:32,166 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=13, startTime=1682416592126; duration=0sec 2023-04-25 09:56:32,166 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:32,166 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:32,167 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:32,167 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:32,167 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:32,167 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b882a59003014c10b541f821ff369253, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/afaca8477b87428e92b9a224c8e8b358, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/bf133d1362f546cea4fa8fd923af39da] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.1 K 2023-04-25 09:56:32,168 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting b882a59003014c10b541f821ff369253, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1682416589449 2023-04-25 09:56:32,168 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/ddfa75d380e5449394d375f35e93e7f2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ddfa75d380e5449394d375f35e93e7f2 2023-04-25 09:56:32,168 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting afaca8477b87428e92b9a224c8e8b358, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682416589517 2023-04-25 09:56:32,169 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting bf133d1362f546cea4fa8fd923af39da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682416590858 2023-04-25 09:56:32,173 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into ddfa75d380e5449394d375f35e93e7f2(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:32,173 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:32,173 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=13, startTime=1682416592126; duration=0sec 2023-04-25 09:56:32,173 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:32,178 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#611 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:32,189 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/d8e90193707e4845847fcd01c1fec008 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/d8e90193707e4845847fcd01c1fec008 2023-04-25 09:56:32,192 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into d8e90193707e4845847fcd01c1fec008(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:32,193 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:32,193 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=13, startTime=1682416592126; duration=0sec 2023-04-25 09:56:32,193 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:32,216 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:32,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:32,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:32,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:32,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:32,219 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:32,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:32,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:32,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:32,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,222 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:32,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:32,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:32,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:32,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:32,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:32,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:32,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:32,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:32,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:32,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:32,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:32,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:32,224 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:32,224 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:32,243 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304255fb1a1e9002f4a34857c4880f74d9207 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255fb1a1e9002f4a34857c4880f74d9207 2023-04-25 09:56:32,650 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/0ec72ae8864a4e4d884bac814ef7ca78 2023-04-25 09:56:33,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:33,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416653021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416653022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48638 deadline: 1682416653022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,060 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/41017b0fdaa043c695ee5352679d8d51 2023-04-25 09:56:33,069 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/78fccc927b7e4b28b2d95184b148aafc 2023-04-25 09:56:33,074 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/0ec72ae8864a4e4d884bac814ef7ca78 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/0ec72ae8864a4e4d884bac814ef7ca78 2023-04-25 09:56:33,078 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/0ec72ae8864a4e4d884bac814ef7ca78, entries=150, sequenceid=329, filesize=25.4 K 2023-04-25 09:56:33,078 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/41017b0fdaa043c695ee5352679d8d51 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/41017b0fdaa043c695ee5352679d8d51 2023-04-25 09:56:33,081 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/41017b0fdaa043c695ee5352679d8d51, entries=150, sequenceid=329, filesize=11.9 K 2023-04-25 09:56:33,082 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/78fccc927b7e4b28b2d95184b148aafc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/78fccc927b7e4b28b2d95184b148aafc 2023-04-25 09:56:33,086 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/78fccc927b7e4b28b2d95184b148aafc, entries=150, sequenceid=329, filesize=11.9 K 2023-04-25 09:56:33,086 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for fb513dc5b3a62f2dba9b88a4e6de5394 in 862ms, sequenceid=329, compaction requested=false 2023-04-25 09:56:33,086 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:33,086 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6488da[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,090 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,090 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,091 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,092 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,092 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,092 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,096 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,099 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,099 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,099 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,102 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,104 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,106 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,107 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,107 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:33,147 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:33,159 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042512374c55f3d84634aa6ef84ecca58dc0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042512374c55f3d84634aa6ef84ecca58dc0 2023-04-25 09:56:33,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416653163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416653164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416653166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,168 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=345, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/023f3e83ab5341359b2a54425e4c5ddc 2023-04-25 09:56:33,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/24986282551f459f9bbbc7233b16a5e8 2023-04-25 09:56:33,207 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,211 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,212 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,212 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,213 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,214 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,214 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,215 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,217 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,217 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,218 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6260f2[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,221 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,226 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,228 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,228 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,228 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,230 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,232 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:33,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,235 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,235 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,235 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416653267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416653267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416653267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,335 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,339 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,339 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,340 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,342 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,342 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,342 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,342 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,342 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,343 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,344 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,344 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f3315cb[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,346 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,346 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,347 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,348 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,351 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,353 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,353 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,353 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,355 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,357 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,358 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,359 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,359 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:33,459 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,463 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,463 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,463 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,463 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,463 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,463 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,464 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,464 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,464 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,471 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416653470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416653470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416653470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,472 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,473 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,473 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,473 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,473 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31c5d569[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,478 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,478 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,479 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,482 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,482 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,484 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,485 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,491 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,491 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,491 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,492 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/84eb1343f4374c8f92cfa4c55cfe4e2f 2023-04-25 09:56:33,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/023f3e83ab5341359b2a54425e4c5ddc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/023f3e83ab5341359b2a54425e4c5ddc 2023-04-25 09:56:33,592 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/023f3e83ab5341359b2a54425e4c5ddc, entries=150, sequenceid=345, filesize=25.4 K 2023-04-25 09:56:33,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/24986282551f459f9bbbc7233b16a5e8 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/24986282551f459f9bbbc7233b16a5e8 2023-04-25 09:56:33,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,596 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,597 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,599 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,599 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,599 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,599 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,601 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,601 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/24986282551f459f9bbbc7233b16a5e8, entries=150, sequenceid=345, filesize=11.9 K 2023-04-25 09:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/84eb1343f4374c8f92cfa4c55cfe4e2f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/84eb1343f4374c8f92cfa4c55cfe4e2f 2023-04-25 09:56:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e08b9d3[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,603 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,605 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,605 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/84eb1343f4374c8f92cfa4c55cfe4e2f, entries=150, sequenceid=345, filesize=11.9 K 2023-04-25 09:56:33,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for fb513dc5b3a62f2dba9b88a4e6de5394 in 461ms, sequenceid=345, compaction requested=true 2023-04-25 09:56:33,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:33,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:33,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:33,607 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:33,607 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:33,607 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,607 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,608 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:33,608 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:33,608 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:33,608 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,608 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,608 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,608 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a1d15e9e437b4fab86cfb4b17e2c8334, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/41017b0fdaa043c695ee5352679d8d51, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/24986282551f459f9bbbc7233b16a5e8] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.3 K 2023-04-25 09:56:33,608 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ddfa75d380e5449394d375f35e93e7f2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/0ec72ae8864a4e4d884bac814ef7ca78, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/023f3e83ab5341359b2a54425e4c5ddc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=76.9 K 2023-04-25 09:56:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,609 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a1d15e9e437b4fab86cfb4b17e2c8334, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682416590858 2023-04-25 09:56:33,609 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 41017b0fdaa043c695ee5352679d8d51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682416590881 2023-04-25 09:56:33,609 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting ddfa75d380e5449394d375f35e93e7f2, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682416590858 2023-04-25 09:56:33,610 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 24986282551f459f9bbbc7233b16a5e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682416593021 2023-04-25 09:56:33,610 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 0ec72ae8864a4e4d884bac814ef7ca78, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682416590881 2023-04-25 09:56:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,610 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 023f3e83ab5341359b2a54425e4c5ddc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682416593021 2023-04-25 09:56:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,611 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,612 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,612 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,612 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,612 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,614 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,615 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,616 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,616 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#618 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,618 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,618 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,619 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,619 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,619 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:33,619 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,620 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,625 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#619 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:33,636 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/985f1e34a251451eb5e2bd085b36b169 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/985f1e34a251451eb5e2bd085b36b169 2023-04-25 09:56:33,641 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into 985f1e34a251451eb5e2bd085b36b169(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:33,642 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:33,642 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=13, startTime=1682416593607; duration=0sec 2023-04-25 09:56:33,642 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:33,642 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:33,643 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:33,643 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:33,643 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,643 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/d8e90193707e4845847fcd01c1fec008, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/78fccc927b7e4b28b2d95184b148aafc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/84eb1343f4374c8f92cfa4c55cfe4e2f] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.3 K 2023-04-25 09:56:33,643 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d8e90193707e4845847fcd01c1fec008, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1682416590858 2023-04-25 09:56:33,644 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 78fccc927b7e4b28b2d95184b148aafc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682416590881 2023-04-25 09:56:33,644 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 84eb1343f4374c8f92cfa4c55cfe4e2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682416593021 2023-04-25 09:56:33,651 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#620 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:33,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:33,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-25 09:56:33,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:33,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:33,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:33,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:33,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:33,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:33,657 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/819cd86bb0934c3f802dc0848feac982 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/819cd86bb0934c3f802dc0848feac982 2023-04-25 09:56:33,662 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into 819cd86bb0934c3f802dc0848feac982(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:33,662 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:33,662 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=13, startTime=1682416593607; duration=0sec 2023-04-25 09:56:33,662 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:33,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416653677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416653677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,719 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,723 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,723 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,724 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,724 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,725 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,726 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,726 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,728 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a32710[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,730 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,731 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,731 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,731 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,732 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,733 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,733 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,733 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,733 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,735 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,736 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,736 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,737 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,739 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,740 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,743 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,743 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,743 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,743 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416653773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416653775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416653775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416653779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416653779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,843 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,847 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,847 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,848 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,848 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,849 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,849 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,850 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,850 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,851 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,851 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,852 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,853 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,853 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e9e582[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,854 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,855 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,855 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,855 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,857 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,857 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,857 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,857 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,859 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,860 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,860 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,861 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,861 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,862 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,867 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,867 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:33,867 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,868 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:33,968 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:33,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:33,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:33,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:33,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:33,971 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:33,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:33,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:33,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,971 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,974 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,974 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:33,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:33,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:33,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:33,975 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:33,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:33,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:33,976 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:33,976 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:33,977 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,977 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58181e8b[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:33,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:33,979 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,980 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,981 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:33,981 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:33,981 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:33,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,981 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:33,982 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:33,982 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,982 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:33,982 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:33,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:33,985 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:33,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:33,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:33,985 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:33,985 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:33,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416653984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:33,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416653984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,987 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:33,988 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:33,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,988 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,990 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:33,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:33,992 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:33,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:33,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:33,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:33,993 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:33,992 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:33,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:33,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:33,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:33,994 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:33,994 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:34,066 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/804c304d15f3464d85bf61336f2ffbb7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/804c304d15f3464d85bf61336f2ffbb7 2023-04-25 09:56:34,070 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into 804c304d15f3464d85bf61336f2ffbb7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:34,070 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:34,071 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=13, startTime=1682416593607; duration=0sec 2023-04-25 09:56:34,071 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:34,076 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042562f6207c968a4cccaf37921ff73c6a4a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042562f6207c968a4cccaf37921ff73c6a4a 2023-04-25 09:56:34,094 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:34,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:34,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:34,096 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:34,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:34,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:34,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,097 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,098 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,098 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:34,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:34,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:34,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:34,099 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:34,099 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,100 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:34,100 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:34,100 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:34,101 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,101 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,102 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:34,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c5e27a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:34,103 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:34,103 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,105 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:34,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,105 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:34,106 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:34,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,107 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:34,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:34,109 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:34,109 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:34,109 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:34,109 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:34,109 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:34,111 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,111 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,111 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,112 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,113 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:34,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,115 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,115 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,115 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:34,115 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,116 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,216 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:34,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:34,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:34,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:34,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:34,219 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:34,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:34,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:34,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,219 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,221 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,221 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,222 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:34,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:34,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:34,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:34,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:34,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:34,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:34,223 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:34,223 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:34,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,224 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,225 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ad7c6b5[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:34,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:34,226 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,227 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:34,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,227 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:34,228 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:34,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,228 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:34,231 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:34,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:34,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:34,231 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:34,231 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:34,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,233 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,234 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,235 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,237 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,237 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:34,237 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:34,237 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:34,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:34,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416654277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:34,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416654277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:34,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416654278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:34,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416654289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:34,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416654289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,337 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:34,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:34,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:34,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:34,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:34,341 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:34,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:34,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:34,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,341 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,345 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,345 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:34,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:34,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:34,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:34,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:34,347 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:34,355 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:34,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@421aff98[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:34,357 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:34,357 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,358 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:34,359 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,359 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:34,359 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,359 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,359 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,360 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:34,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:34,362 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:34,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:34,362 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:34,363 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:34,363 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:34,364 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,364 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,364 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,365 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,366 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,372 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:34,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,372 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,373 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:34,373 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:34,373 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,473 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:34,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:34,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:34,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:34,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:34,476 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:34,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:34,476 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:34,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,480 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=363, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/1f3c873876574bc0b277061c4ba1feba 2023-04-25 09:56:34,480 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:34,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:34,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:34,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:34,483 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:34,485 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:34,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aee5d16[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:34,489 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:34,489 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/f5db3a865fa54eb6920b75aa31807d98 2023-04-25 09:56:34,490 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:34,491 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:34,491 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,491 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:34,493 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:34,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,493 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:34,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:34,496 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:34,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:34,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:34,496 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:34,496 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:34,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,498 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,499 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/e14c34298d4b4ac696eece5892df6cf9 2023-04-25 09:56:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,501 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,504 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:34,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:34,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/1f3c873876574bc0b277061c4ba1feba as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/1f3c873876574bc0b277061c4ba1feba 2023-04-25 09:56:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,505 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,505 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:34,505 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:34,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/1f3c873876574bc0b277061c4ba1feba, entries=200, sequenceid=363, filesize=32.3 K 2023-04-25 09:56:34,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/f5db3a865fa54eb6920b75aa31807d98 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f5db3a865fa54eb6920b75aa31807d98 2023-04-25 09:56:34,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f5db3a865fa54eb6920b75aa31807d98, entries=150, sequenceid=363, filesize=11.9 K 2023-04-25 09:56:34,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/e14c34298d4b4ac696eece5892df6cf9 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e14c34298d4b4ac696eece5892df6cf9 2023-04-25 09:56:34,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e14c34298d4b4ac696eece5892df6cf9, entries=150, sequenceid=363, filesize=11.9 K 2023-04-25 09:56:34,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for fb513dc5b3a62f2dba9b88a4e6de5394 in 862ms, sequenceid=363, compaction requested=false 2023-04-25 09:56:34,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:34,605 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:34,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:34,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:34,609 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:34,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:34,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:34,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,609 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,611 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,611 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:34,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:34,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:34,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,612 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:34,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:34,613 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:34,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:34,613 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-25 09:56:34,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:34,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:34,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:34,613 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:34,614 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:34,614 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:34,625 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042523f0e45344d74d3fb57b9fd8312e2c66 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042523f0e45344d74d3fb57b9fd8312e2c66 2023-04-25 09:56:34,629 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=380, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/bb193bbbd5d04672949d36c1e52e51b6 2023-04-25 09:56:34,637 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/7528c01a456642d38155dbb579fe8b99 2023-04-25 09:56:34,651 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/02eb04aed04343aca3de204c15d96c56 2023-04-25 09:56:34,655 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/bb193bbbd5d04672949d36c1e52e51b6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/bb193bbbd5d04672949d36c1e52e51b6 2023-04-25 09:56:34,659 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/bb193bbbd5d04672949d36c1e52e51b6, entries=150, sequenceid=380, filesize=25.4 K 2023-04-25 09:56:34,660 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/7528c01a456642d38155dbb579fe8b99 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7528c01a456642d38155dbb579fe8b99 2023-04-25 09:56:34,668 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7528c01a456642d38155dbb579fe8b99, entries=150, sequenceid=380, filesize=11.9 K 2023-04-25 09:56:34,673 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/02eb04aed04343aca3de204c15d96c56 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/02eb04aed04343aca3de204c15d96c56 2023-04-25 09:56:34,676 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/02eb04aed04343aca3de204c15d96c56, entries=150, sequenceid=380, filesize=11.9 K 2023-04-25 09:56:34,677 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 64ms, sequenceid=380, compaction requested=true 2023-04-25 09:56:34,677 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:34,677 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:34,677 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:34,682 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,682 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:34,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d4b203[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:34,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:34,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,685 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:34,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:34,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:34,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:34,689 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:34,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:34,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:34,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:34,689 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:34,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,691 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:34,692 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:34,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:34,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:34,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:34,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:34,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:34,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:34,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:34,696 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:34,697 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:34,797 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:34,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:34,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:34,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:34,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:34,800 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:34,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:34,800 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:34,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,801 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,802 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:34,802 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:34,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:34,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:34,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:34,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:34,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:34,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:34,804 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:34,804 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:34,804 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:34,804 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-25 09:56:34,804 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:34,805 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:34,805 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:34,805 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:34,805 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:34,805 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:34,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:34,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:34,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416594840, max timestamp=1682416594849Num uniques 859; 2023-04-25 09:56:34,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:34,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416594840, max timestamp=1682416594849Num uniques 859; 2023-04-25 09:56:34,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416594840, max timestamp=1682416594849Num uniques 859; 2023-04-25 09:56:34,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:34,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:34,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416594869, max timestamp=1682416594876Num uniques 1018; 2023-04-25 09:56:34,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416594869, max timestamp=1682416594876Num uniques 1018; 2023-04-25 09:56:34,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:56:34,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682416594869, max timestamp=1682416594876Num uniques 1018; 2023-04-25 09:56:34,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:56:34,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-25 09:56:34,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682416594898, max timestamp=1682416594903Num uniques 1077; 2023-04-25 09:56:34,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-25 09:56:34,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416594898, max timestamp=1682416594903Num uniques 1077; 2023-04-25 09:56:34,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-25 09:56:34,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416594898, max timestamp=1682416594903Num uniques 1077; 2023-04-25 09:56:34,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-25 09:56:34,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682416594919, max timestamp=1682416594928Num uniques 1036; 2023-04-25 09:56:34,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682416594919, max timestamp=1682416594928Num uniques 1009; 2023-04-25 09:56:34,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682416594919, max timestamp=1682416594928Num uniques 1009; 2023-04-25 09:56:34,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-25 09:56:34,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-25 09:56:34,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-25 09:56:34,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416594952, max timestamp=1682416594961Num uniques 1095; 2023-04-25 09:56:34,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-25 09:56:34,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416594952, max timestamp=1682416594961Num uniques 1045; 2023-04-25 09:56:34,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-25 09:56:34,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682416594952, max timestamp=1682416594961Num uniques 1045; 2023-04-25 09:56:34,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-25 09:56:34,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682416594977, max timestamp=1682416594992Num uniques 1054; 2023-04-25 09:56:34,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 09:56:34,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682416594977, max timestamp=1682416594992Num uniques 1054; 2023-04-25 09:56:34,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682416594977, max timestamp=1682416594992Num uniques 1054; 2023-04-25 09:56:34,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 09:56:34,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-25 09:56:35,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682416595006, max timestamp=1682416595023Num uniques 1063; 2023-04-25 09:56:35,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682416595006, max timestamp=1682416595023Num uniques 1063; 2023-04-25 09:56:35,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682416595006, max timestamp=1682416595023Num uniques 1063; 2023-04-25 09:56:35,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-25 09:56:35,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-25 09:56:35,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-25 09:56:35,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682416595059, max timestamp=1682416595075Num uniques 1022; 2023-04-25 09:56:35,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682416595059, max timestamp=1682416595075Num uniques 1022; 2023-04-25 09:56:35,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-25 09:56:35,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682416595059, max timestamp=1682416595075Num uniques 1022; 2023-04-25 09:56:35,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-25 09:56:35,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-25 09:56:35,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682416595094, max timestamp=1682416595116Num uniques 1081; 2023-04-25 09:56:35,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 09:56:35,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682416595094, max timestamp=1682416595116Num uniques 1081; 2023-04-25 09:56:35,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682416595094, max timestamp=1682416595116Num uniques 1081; 2023-04-25 09:56:35,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 09:56:35,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-25 09:56:35,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682416595121, max timestamp=1682416595143Num uniques 1040; 2023-04-25 09:56:35,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-25 09:56:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682416595121, max timestamp=1682416595143Num uniques 1040; 2023-04-25 09:56:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-25 09:56:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682416595121, max timestamp=1682416595143Num uniques 1040; 2023-04-25 09:56:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-25 09:56:35,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682416595162, max timestamp=1682416595171Num uniques 1049; 2023-04-25 09:56:35,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-25 09:56:35,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682416595162, max timestamp=1682416595171Num uniques 1049; 2023-04-25 09:56:35,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682416595162, max timestamp=1682416595171Num uniques 1049; 2023-04-25 09:56:35,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-25 09:56:35,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-25 09:56:35,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682416595196, max timestamp=1682416595205Num uniques 1058; 2023-04-25 09:56:35,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682416595196, max timestamp=1682416595205Num uniques 1058; 2023-04-25 09:56:35,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-25 09:56:35,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416595197, max timestamp=1682416595205Num uniques 1058; 2023-04-25 09:56:35,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=6870, suffix heap size=138000,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=150 2023-04-25 09:56:35,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-25 09:56:35,218 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425af4b059d0c8c43a5aeb185f5c5490cdc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425af4b059d0c8c43a5aeb185f5c5490cdc 2023-04-25 09:56:35,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682416595226, max timestamp=1682416595238Num uniques 1067; 2023-04-25 09:56:35,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682416595226, max timestamp=1682416595238Num uniques 1067; 2023-04-25 09:56:35,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-25 09:56:35,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-25 09:56:35,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682416595226, max timestamp=1682416595238Num uniques 1009; 2023-04-25 09:56:35,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=9932, suffix heap size=132904,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=217 2023-04-25 09:56:35,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682416595258, max timestamp=1682416595270Num uniques 1076; 2023-04-25 09:56:35,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-25 09:56:35,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682416595258, max timestamp=1682416595270Num uniques 1076; 2023-04-25 09:56:35,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-25 09:56:35,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682416595258, max timestamp=1682416595270Num uniques 1076; 2023-04-25 09:56:35,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-25 09:56:35,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1682416595285, max timestamp=1682416595292Num uniques 1035; 2023-04-25 09:56:35,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8464, totalHeapSize=19480, min timestamp=1682416595285, max timestamp=1682416595292Num uniques 1035; 2023-04-25 09:56:35,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8461, totalHeapSize=19480, min timestamp=1682416595285, max timestamp=1682416595292Num uniques 1035; 2023-04-25 09:56:35,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47391, new segment data size=8461, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-25 09:56:35,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47390, new segment data size=8460, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-25 09:56:35,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47394, new segment data size=8464, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-25 09:56:35,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1682416595301, max timestamp=1682416595308Num uniques 1044; 2023-04-25 09:56:35,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1682416595301, max timestamp=1682416595308Num uniques 1044; 2023-04-25 09:56:35,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=15754, suffix heap size=136544,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=344 2023-04-25 09:56:35,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=15754, suffix heap size=136544,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=344 2023-04-25 09:56:35,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1682416595301, max timestamp=1682416595308Num uniques 1044; 2023-04-25 09:56:35,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=15754, suffix heap size=136544,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=344 2023-04-25 09:56:35,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416655324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1682416595318, max timestamp=1682416595325Num uniques 1203; 2023-04-25 09:56:35,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416655325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55088, new segment data size=9288, suffix heap size=153080,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=203 2023-04-25 09:56:35,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1682416595318, max timestamp=1682416595325Num uniques 1203; 2023-04-25 09:56:35,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=55088, new segment data size=9288, suffix heap size=153080,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=203 2023-04-25 09:56:35,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9290, totalHeapSize=21352, min timestamp=1682416595318, max timestamp=1682416595325Num uniques 1203; 2023-04-25 09:56:35,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=55090, new segment data size=9290, suffix heap size=153080,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=203 2023-04-25 09:56:35,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682416595340, max timestamp=1682416595347Num uniques 1062; 2023-04-25 09:56:35,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-25 09:56:35,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682416595340, max timestamp=1682416595347Num uniques 1062; 2023-04-25 09:56:35,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-25 09:56:35,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682416595340, max timestamp=1682416595347Num uniques 1062; 2023-04-25 09:56:35,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-04-25 09:56:35,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1682416595360, max timestamp=1682416595376Num uniques 1071; 2023-04-25 09:56:35,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1682416595360, max timestamp=1682416595376Num uniques 1071; 2023-04-25 09:56:35,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-25 09:56:35,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1682416595360, max timestamp=1682416595376Num uniques 1071; 2023-04-25 09:56:35,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-25 09:56:35,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-25 09:56:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1682416595393, max timestamp=1682416595403Num uniques 1130; 2023-04-25 09:56:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682416595393, max timestamp=1682416595403Num uniques 1130; 2023-04-25 09:56:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=12820, suffix heap size=145488,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=280 2023-04-25 09:56:35,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-25 09:56:35,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682416595393, max timestamp=1682416595403Num uniques 1130; 2023-04-25 09:56:35,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-25 09:56:35,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682416595418, max timestamp=1682416595432Num uniques 1139; 2023-04-25 09:56:35,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=8654, suffix heap size=146424,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=189 2023-04-25 09:56:35,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682416595418, max timestamp=1682416595432Num uniques 1039; 2023-04-25 09:56:35,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-25 09:56:35,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682416595418, max timestamp=1682416595432Num uniques 1039; 2023-04-25 09:56:35,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-25 09:56:35,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13638, totalHeapSize=31232, min timestamp=1682416595442, max timestamp=1682416595450Num uniques 1048; 2023-04-25 09:56:35,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=13638, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-25 09:56:35,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13638, totalHeapSize=31232, min timestamp=1682416595442, max timestamp=1682416595450Num uniques 1048; 2023-04-25 09:56:35,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=13638, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-25 09:56:35,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13638, totalHeapSize=31232, min timestamp=1682416595442, max timestamp=1682416595450Num uniques 1048; 2023-04-25 09:56:35,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=13638, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-25 09:56:35,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682416595466, max timestamp=1682416595472Num uniques 1157; 2023-04-25 09:56:35,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=9475, suffix heap size=148296,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=207 2023-04-25 09:56:35,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682416595466, max timestamp=1682416595472Num uniques 1157; 2023-04-25 09:56:35,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682416595466, max timestamp=1682416595472Num uniques 1157; 2023-04-25 09:56:35,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=9475, suffix heap size=148296,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=207 2023-04-25 09:56:35,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=9475, suffix heap size=148296,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=207 2023-04-25 09:56:35,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682416595485, max timestamp=1682416595489Num uniques 1066; 2023-04-25 09:56:35,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-25 09:56:35,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682416595485, max timestamp=1682416595489Num uniques 1066; 2023-04-25 09:56:35,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682416595485, max timestamp=1682416595489Num uniques 1066; 2023-04-25 09:56:35,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-25 09:56:35,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-25 09:56:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1682416595502, max timestamp=1682416595507Num uniques 1075; 2023-04-25 09:56:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-25 09:56:35,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1682416595502, max timestamp=1682416595507Num uniques 1075; 2023-04-25 09:56:35,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-25 09:56:35,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1682416595502, max timestamp=1682416595507Num uniques 1075; 2023-04-25 09:56:35,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-04-25 09:56:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1682416595514, max timestamp=1682416595526Num uniques 1084; 2023-04-25 09:56:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-25 09:56:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1682416595514, max timestamp=1682416595526Num uniques 1084; 2023-04-25 09:56:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-25 09:56:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416595514, max timestamp=1682416595526Num uniques 1084; 2023-04-25 09:56:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=6870, suffix heap size=140704,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=150 2023-04-25 09:56:35,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682416595540, max timestamp=1682416595545Num uniques 1043; 2023-04-25 09:56:35,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682416595540, max timestamp=1682416595545Num uniques 1043; 2023-04-25 09:56:35,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-25 09:56:35,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416595540, max timestamp=1682416595545Num uniques 1009; 2023-04-25 09:56:35,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-25 09:56:35,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=11128, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-25 09:56:35,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=352, cellSize=16120, totalHeapSize=36848, min timestamp=1682416595551, max timestamp=1682416595565Num uniques 1102; 2023-04-25 09:56:35,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=16120, suffix heap size=142576,new segment heap size=36848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=352 2023-04-25 09:56:35,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=352, cellSize=16120, totalHeapSize=36848, min timestamp=1682416595551, max timestamp=1682416595565Num uniques 1102; 2023-04-25 09:56:35,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=352, cellSize=16120, totalHeapSize=36848, min timestamp=1682416595551, max timestamp=1682416595565Num uniques 1009; 2023-04-25 09:56:35,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=16120, suffix heap size=142576,new segment heap size=36848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=352 2023-04-25 09:56:35,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=16120, suffix heap size=132904,new segment heap size=36848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=352 2023-04-25 09:56:35,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1682416595578, max timestamp=1682416595586Num uniques 1211; 2023-04-25 09:56:35,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1682416595578, max timestamp=1682416595586Num uniques 1211; 2023-04-25 09:56:35,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55456, new segment data size=9656, suffix heap size=153912,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1211, new segment cells count=211 2023-04-25 09:56:35,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1682416595582, max timestamp=1682416595586Num uniques 1211; 2023-04-25 09:56:35,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=55456, new segment data size=7366, suffix heap size=153912,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1211, new segment cells count=161 2023-04-25 09:56:35,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55456, new segment data size=9656, suffix heap size=153912,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1211, new segment cells count=211 2023-04-25 09:56:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1682416595598, max timestamp=1682416595603Num uniques 1070; 2023-04-25 09:56:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1682416595598, max timestamp=1682416595603Num uniques 1070; 2023-04-25 09:56:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1682416595598, max timestamp=1682416595603Num uniques 1020; 2023-04-25 09:56:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=12360, suffix heap size=139248,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=270 2023-04-25 09:56:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=12360, suffix heap size=134048,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=270 2023-04-25 09:56:35,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=12360, suffix heap size=139248,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=270 2023-04-25 09:56:35,627 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/e3a19034c96d47c9888913fdecd9e0dc 2023-04-25 09:56:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682416595618, max timestamp=1682416595627Num uniques 1129; 2023-04-25 09:56:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-04-25 09:56:35,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682416595618, max timestamp=1682416595627Num uniques 1129; 2023-04-25 09:56:35,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1682416595618, max timestamp=1682416595627Num uniques 1129; 2023-04-25 09:56:35,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-04-25 09:56:35,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=10484, suffix heap size=145384,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=229 2023-04-25 09:56:35,639 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/1c56b6be557c4c84a28297199295fdb2 2023-04-25 09:56:35,647 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/e7f7b2ccca764ca8b72f35ed307cd975 2023-04-25 09:56:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1682416595634, max timestamp=1682416595649Num uniques 1088; 2023-04-25 09:56:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1682416595634, max timestamp=1682416595649Num uniques 1088; 2023-04-25 09:56:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=8608, suffix heap size=141120,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=188 2023-04-25 09:56:35,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=8608, suffix heap size=141120,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=188 2023-04-25 09:56:35,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682416595634, max timestamp=1682416595649Num uniques 1088; 2023-04-25 09:56:35,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=6870, suffix heap size=141120,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=150 2023-04-25 09:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/e3a19034c96d47c9888913fdecd9e0dc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e3a19034c96d47c9888913fdecd9e0dc 2023-04-25 09:56:35,666 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e3a19034c96d47c9888913fdecd9e0dc, entries=100, sequenceid=385, filesize=18.6 K 2023-04-25 09:56:35,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/1c56b6be557c4c84a28297199295fdb2 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/1c56b6be557c4c84a28297199295fdb2 2023-04-25 09:56:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1682416595662, max timestamp=1682416595668Num uniques 1047; 2023-04-25 09:56:35,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1682416595662, max timestamp=1682416595668Num uniques 1047; 2023-04-25 09:56:35,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=13602, suffix heap size=136856,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=297 2023-04-25 09:56:35,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=13602, suffix heap size=136856,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=297 2023-04-25 09:56:35,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=297, cellSize=13602, totalHeapSize=31128, min timestamp=1682416595662, max timestamp=1682416595668Num uniques 1009; 2023-04-25 09:56:35,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13602, suffix heap size=132904,new segment heap size=31128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=297 2023-04-25 09:56:35,675 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/1c56b6be557c4c84a28297199295fdb2, entries=50, sequenceid=385, filesize=7.1 K 2023-04-25 09:56:35,676 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/e7f7b2ccca764ca8b72f35ed307cd975 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e7f7b2ccca764ca8b72f35ed307cd975 2023-04-25 09:56:35,680 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e7f7b2ccca764ca8b72f35ed307cd975, entries=50, sequenceid=385, filesize=7.1 K 2023-04-25 09:56:35,681 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=93.93 KB/96180 for fb513dc5b3a62f2dba9b88a4e6de5394 in 877ms, sequenceid=385, compaction requested=true 2023-04-25 09:56:35,681 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:35,681 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:35,681 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:35,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:35,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=107.34 KB heapSize=292.36 KB 2023-04-25 09:56:35,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:35,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:35,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:35,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:35,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:35,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:35,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c22802[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:35,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:35,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,692 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:35,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:35,692 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:35,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:35,695 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:35,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:35,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416655693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:35,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:35,699 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:35,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:35,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:35,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:35,699 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:35,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416655694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416655694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416655695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,701 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416655698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,703 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d9004c584bc94787aa6f60fb832c55a9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d9004c584bc94787aa6f60fb832c55a9 2023-04-25 09:56:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:35,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:35,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:35,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:35,707 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:35,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:35,708 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:35,707 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,708 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:35,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:35,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416655799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416655801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416655801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416655801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:35,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416655804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,808 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:35,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:35,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:35,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:35,811 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:35,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:35,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:35,811 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:35,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,812 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,813 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:35,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:35,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:35,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:35,815 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:35,815 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:35,815 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:35,815 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:35,815 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:35,818 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,818 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:35,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b1f9fd[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:35,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:35,820 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:35,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:35,821 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:35,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:35,822 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,822 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,823 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:35,825 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:35,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:35,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:35,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:35,825 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:35,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,827 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,828 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,829 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:35,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:35,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:35,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:35,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:35,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:35,832 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,832 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:35,833 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,833 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:35,833 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:35,933 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:35,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:35,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:35,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:35,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:35,937 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,937 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,938 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,938 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:35,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:35,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:35,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:35,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:35,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:35,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:35,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:35,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:35,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:35,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:35,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:35,940 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:35,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:35,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:35,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:35,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:35,942 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,942 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9f08f0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:35,950 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:35,950 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,951 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:35,951 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:35,951 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,951 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:35,954 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:35,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,956 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:35,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:35,958 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:35,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:35,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:35,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:35,958 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:35,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,960 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:35,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:35,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,961 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:35,963 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:35,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:35,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:35,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:35,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:35,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:35,965 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:35,966 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:35,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:35,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:35,966 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,966 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:35,966 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:35,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:35,966 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:35,966 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416656002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416656003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416656004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416656004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416656006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,066 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,069 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,070 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,071 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,071 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,072 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,073 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,073 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:36,073 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,076 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,076 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc8f9aa[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,077 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,078 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,079 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,082 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,085 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,085 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,085 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,088 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,088 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,090 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,091 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,092 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,092 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,091 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,092 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:36,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,118 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=965, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/4b802a1af5834b27b40aefb4f22b754b 2023-04-25 09:56:36,192 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,195 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,195 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,196 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,197 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,200 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,200 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,201 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,201 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,201 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:36,201 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,201 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,202 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,202 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,203 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f195e09[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,204 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,206 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,206 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,206 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,207 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,207 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,209 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,210 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,210 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,211 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,220 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,221 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,223 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,223 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,224 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,224 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,224 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416656305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416656308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416656308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416656308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416656311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,324 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,327 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,327 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,329 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,330 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,331 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,331 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,331 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:36,331 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,331 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,332 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,333 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673f2606[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,335 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,336 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,336 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,336 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,337 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,340 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,340 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,340 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,346 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,347 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,347 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,349 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,352 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,352 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,353 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,353 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:36,453 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,456 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,457 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,458 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,458 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,459 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,460 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,460 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,460 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,460 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:36,460 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,461 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,462 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,462 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e06a99[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,464 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,464 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,466 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,466 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,468 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,470 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,470 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,470 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,470 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,471 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,477 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:36,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,481 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,481 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=965 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/a8042e3a45bb4376bd284d204b4236db 2023-04-25 09:56:36,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=965 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/f7fab86679b546e9bb8c7c0957174af3 2023-04-25 09:56:36,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/4b802a1af5834b27b40aefb4f22b754b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/4b802a1af5834b27b40aefb4f22b754b 2023-04-25 09:56:36,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/4b802a1af5834b27b40aefb4f22b754b, entries=150, sequenceid=965, filesize=25.4 K 2023-04-25 09:56:36,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/a8042e3a45bb4376bd284d204b4236db as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a8042e3a45bb4376bd284d204b4236db 2023-04-25 09:56:36,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a8042e3a45bb4376bd284d204b4236db, entries=150, sequenceid=965, filesize=11.9 K 2023-04-25 09:56:36,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/f7fab86679b546e9bb8c7c0957174af3 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f7fab86679b546e9bb8c7c0957174af3 2023-04-25 09:56:36,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f7fab86679b546e9bb8c7c0957174af3, entries=150, sequenceid=965, filesize=11.9 K 2023-04-25 09:56:36,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~292.31 KB/299328, currentSize=73.80 KB/75570 for fb513dc5b3a62f2dba9b88a4e6de5394 in 884ms, sequenceid=965, compaction requested=true 2023-04-25 09:56:36,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:36,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:36,566 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:56:36,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:36,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:56:36,566 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:56:36,568 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130970 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:56:36,568 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:36,568 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,568 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/819cd86bb0934c3f802dc0848feac982, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/1f3c873876574bc0b277061c4ba1feba, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/bb193bbbd5d04672949d36c1e52e51b6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e3a19034c96d47c9888913fdecd9e0dc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/4b802a1af5834b27b40aefb4f22b754b] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=127.9 K 2023-04-25 09:56:36,569 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56794 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:56:36,569 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:36,569 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,569 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/985f1e34a251451eb5e2bd085b36b169, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f5db3a865fa54eb6920b75aa31807d98, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7528c01a456642d38155dbb579fe8b99, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/1c56b6be557c4c84a28297199295fdb2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a8042e3a45bb4376bd284d204b4236db] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=55.5 K 2023-04-25 09:56:36,569 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 819cd86bb0934c3f802dc0848feac982, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682416593021 2023-04-25 09:56:36,569 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 985f1e34a251451eb5e2bd085b36b169, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682416593021 2023-04-25 09:56:36,569 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 1f3c873876574bc0b277061c4ba1feba, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682416593154 2023-04-25 09:56:36,570 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting f5db3a865fa54eb6920b75aa31807d98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682416593163 2023-04-25 09:56:36,570 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting bb193bbbd5d04672949d36c1e52e51b6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1682416593669 2023-04-25 09:56:36,570 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7528c01a456642d38155dbb579fe8b99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1682416593669 2023-04-25 09:56:36,571 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e3a19034c96d47c9888913fdecd9e0dc, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682416594802 2023-04-25 09:56:36,571 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 1c56b6be557c4c84a28297199295fdb2, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682416594803 2023-04-25 09:56:36,571 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a8042e3a45bb4376bd284d204b4236db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=965, earliestPutTs=1682416595675 2023-04-25 09:56:36,571 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 4b802a1af5834b27b40aefb4f22b754b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=965, earliestPutTs=1682416595675 2023-04-25 09:56:36,580 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#633 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:36,581 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,596 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,596 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,597 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,598 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,598 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,599 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#634 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,600 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:36,600 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:36,628 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425d878255c87ca4c8081c74bcb09b1e049 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d878255c87ca4c8081c74bcb09b1e049 2023-04-25 09:56:36,634 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=979, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/64d80a3805ee496ab49008c2c354c341 2023-04-25 09:56:36,637 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/660b13f8c2004c51acdda1c823cf68d7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/660b13f8c2004c51acdda1c823cf68d7 2023-04-25 09:56:36,643 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into 660b13f8c2004c51acdda1c823cf68d7(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:36,643 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:36,643 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=11, startTime=1682416596566; duration=0sec 2023-04-25 09:56:36,644 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:36,644 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-25 09:56:36,645 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/261ede963f134021b489055483e3357a 2023-04-25 09:56:36,645 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56794 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-25 09:56:36,645 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:36,645 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,645 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/804c304d15f3464d85bf61336f2ffbb7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e14c34298d4b4ac696eece5892df6cf9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/02eb04aed04343aca3de204c15d96c56, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e7f7b2ccca764ca8b72f35ed307cd975, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f7fab86679b546e9bb8c7c0957174af3] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=55.5 K 2023-04-25 09:56:36,646 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 804c304d15f3464d85bf61336f2ffbb7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1682416593021 2023-04-25 09:56:36,646 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e14c34298d4b4ac696eece5892df6cf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1682416593163 2023-04-25 09:56:36,646 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 02eb04aed04343aca3de204c15d96c56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1682416593669 2023-04-25 09:56:36,647 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting e7f7b2ccca764ca8b72f35ed307cd975, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682416594803 2023-04-25 09:56:36,647 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting f7fab86679b546e9bb8c7c0957174af3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=965, earliestPutTs=1682416595675 2023-04-25 09:56:36,653 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/fae9049c28704c2899b65e956aaa9ad4 2023-04-25 09:56:36,656 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:36,659 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/64d80a3805ee496ab49008c2c354c341 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/64d80a3805ee496ab49008c2c354c341 2023-04-25 09:56:36,661 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/64d80a3805ee496ab49008c2c354c341, entries=150, sequenceid=979, filesize=25.4 K 2023-04-25 09:56:36,663 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/261ede963f134021b489055483e3357a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/261ede963f134021b489055483e3357a 2023-04-25 09:56:36,666 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/261ede963f134021b489055483e3357a, entries=150, sequenceid=979, filesize=11.9 K 2023-04-25 09:56:36,667 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/fae9049c28704c2899b65e956aaa9ad4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fae9049c28704c2899b65e956aaa9ad4 2023-04-25 09:56:36,667 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/7bdf5efe68c745fdaa05abc5a561cb7c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/7bdf5efe68c745fdaa05abc5a561cb7c 2023-04-25 09:56:36,671 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fae9049c28704c2899b65e956aaa9ad4, entries=150, sequenceid=979, filesize=11.9 K 2023-04-25 09:56:36,671 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 71ms, sequenceid=979, compaction requested=false 2023-04-25 09:56:36,671 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:36,672 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,672 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,676 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,678 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into 7bdf5efe68c745fdaa05abc5a561cb7c(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:36,678 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:36,678 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=11, startTime=1682416596566; duration=0sec 2023-04-25 09:56:36,678 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,683 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4eda86[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,685 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,686 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,691 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,691 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,691 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,693 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,693 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,693 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,693 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,695 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,698 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,698 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,699 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,699 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,699 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,799 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,803 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,803 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,804 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,805 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,805 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,806 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:36,807 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,810 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,811 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed22cf0[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,814 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,814 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,816 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,819 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,819 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,819 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,821 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,821 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,823 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,825 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,825 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,826 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,826 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:36,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-25 09:56:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:36,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:36,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:36,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416656850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416656851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416656852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416596841, max timestamp=1682416596853Num uniques 859; 2023-04-25 09:56:36,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416656853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:56:36,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682416596841, max timestamp=1682416596853Num uniques 859; 2023-04-25 09:56:36,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682416596841, max timestamp=1682416596853Num uniques 859; 2023-04-25 09:56:36,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-25 09:56:36,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-25 09:56:36,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416656901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,926 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:36,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:36,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:36,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:36,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:36,929 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:36,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:36,929 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:36,930 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,930 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,931 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,931 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,932 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:36,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:36,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:36,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:36,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:36,933 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:36,936 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c3323a[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:36,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:36,939 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:36,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:36,940 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:36,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,940 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:36,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:36,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:36,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:36,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:36,944 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:36,944 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:36,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,946 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:36,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,947 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,948 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416656956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416656956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:36,958 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:36,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:36,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:36,958 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:36,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:36,958 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:36,958 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:36,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:36,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416656956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:36,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416656957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:36,994 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/532b01cd9e9f415b83352ad9339f882f as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/532b01cd9e9f415b83352ad9339f882f 2023-04-25 09:56:36,998 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into 532b01cd9e9f415b83352ad9339f882f(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:36,998 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:36,998 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=11, startTime=1682416596566; duration=0sec 2023-04-25 09:56:36,998 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:37,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416657005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,059 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:37,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:37,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:37,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:37,062 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:37,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:37,062 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:37,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,063 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,064 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,064 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:37,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:37,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:37,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:37,065 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,066 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:37,066 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:37,066 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:37,067 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,067 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,068 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:37,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43942019[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:37,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:37,069 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,071 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:37,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,072 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:37,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:37,073 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,073 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:37,075 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:37,075 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:37,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:37,076 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:37,076 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:37,077 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,077 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,077 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,078 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,079 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,081 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,081 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:37,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:37,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:37,082 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:37,082 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:37,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416657158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416657158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416657161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416657161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,182 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:37,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:37,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:37,185 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:37,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:37,185 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:37,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,188 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:37,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:37,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:37,189 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,189 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:37,190 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:37,190 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:37,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,191 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,191 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,191 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:37,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e334807[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:37,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:37,193 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,194 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:37,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,194 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:37,195 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:37,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,196 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:37,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:37,198 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:37,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:37,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:37,199 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:37,199 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:37,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,201 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,202 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,203 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,205 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:37,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:37,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:37,205 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,206 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,206 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:37,206 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:37,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416657210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,248 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425f03a4e4461704cc08e611946096df31f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f03a4e4461704cc08e611946096df31f 2023-04-25 09:56:37,306 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:37,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:37,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:37,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:37,309 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:37,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:37,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:37,309 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,310 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,311 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:37,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:37,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:37,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:37,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:37,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:37,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:37,313 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:37,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:37,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:37,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:37,314 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,314 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:37,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51294e9d[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:37,316 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:37,316 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,318 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:37,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:37,318 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:37,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,318 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:37,320 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:37,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:37,322 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:37,323 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:37,323 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:37,323 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:37,323 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:37,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,324 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,325 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,328 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:37,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:37,328 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,329 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,329 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:37,329 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:37,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:37,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,429 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:37,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:37,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:37,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:37,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:37,432 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:37,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:37,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:37,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,432 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,437 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,437 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,438 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,438 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:37,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:37,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:37,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:37,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:37,439 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:37,441 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:37,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6842f460[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:37,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:37,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,444 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:37,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:37,445 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:37,446 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,446 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:37,447 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:37,447 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,447 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,447 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:37,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:37,450 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:37,450 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:37,450 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:37,450 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:37,450 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:37,452 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,452 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,452 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,453 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,454 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,456 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:37,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:37,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,456 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:37,456 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,456 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416657461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416657461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416657465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416657466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416657513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,556 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:37,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:37,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:37,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:37,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:37,560 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:37,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:37,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:37,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,561 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,562 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:37,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:37,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:37,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:37,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:37,564 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:37,567 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,567 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:37,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ab48599[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:37,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:37,569 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,570 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:37,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,570 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:37,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:37,570 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:37,571 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,571 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,572 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:37,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:37,574 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:37,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:37,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:37,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:37,574 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:37,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,576 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:37,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:37,577 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:37,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:37,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:37,582 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:37,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,583 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:37,582 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,583 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:37,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:37,583 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:37,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:37,653 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=993, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/3bf9532a8609455787c711a66d3557a5 2023-04-25 09:56:37,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=993 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/9f3e98239667445e9b63472e3ae31e36 2023-04-25 09:56:37,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=993 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/a4cced7c680a45988c243889b80c5e5a 2023-04-25 09:56:37,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/3bf9532a8609455787c711a66d3557a5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/3bf9532a8609455787c711a66d3557a5 2023-04-25 09:56:37,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/3bf9532a8609455787c711a66d3557a5, entries=150, sequenceid=993, filesize=25.4 K 2023-04-25 09:56:37,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/9f3e98239667445e9b63472e3ae31e36 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/9f3e98239667445e9b63472e3ae31e36 2023-04-25 09:56:37,683 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:37,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:37,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/9f3e98239667445e9b63472e3ae31e36, entries=150, sequenceid=993, filesize=11.9 K 2023-04-25 09:56:37,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/a4cced7c680a45988c243889b80c5e5a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a4cced7c680a45988c243889b80c5e5a 2023-04-25 09:56:37,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:37,686 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:37,686 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:37,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:37,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,687 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,688 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:37,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:37,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a4cced7c680a45988c243889b80c5e5a, entries=150, sequenceid=993, filesize=11.9 K 2023-04-25 09:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,688 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:37,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:37,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for fb513dc5b3a62f2dba9b88a4e6de5394 in 853ms, sequenceid=993, compaction requested=true 2023-04-25 09:56:37,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:37,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:37,689 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:37,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:37,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:37,689 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:37,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:37,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:37,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:37,690 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,690 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:37,690 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:37,690 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-25 09:56:37,690 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:37,691 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:37,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:37,691 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/A is initiating minor compaction (all files) 2023-04-25 09:56:37,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:37,691 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/A in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:37,691 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/660b13f8c2004c51acdda1c823cf68d7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/64d80a3805ee496ab49008c2c354c341, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/3bf9532a8609455787c711a66d3557a5] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=77.2 K 2023-04-25 09:56:37,691 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:37,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:37,691 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/532b01cd9e9f415b83352ad9339f882f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/261ede963f134021b489055483e3357a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/9f3e98239667445e9b63472e3ae31e36] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.6 K 2023-04-25 09:56:37,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:37,691 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 660b13f8c2004c51acdda1c823cf68d7, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=965, earliestPutTs=1682416595675 2023-04-25 09:56:37,691 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-25 09:56:37,692 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 532b01cd9e9f415b83352ad9339f882f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=965, earliestPutTs=1682416595675 2023-04-25 09:56:37,692 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 64d80a3805ee496ab49008c2c354c341, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1682416595691 2023-04-25 09:56:37,692 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 261ede963f134021b489055483e3357a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1682416595691 2023-04-25 09:56:37,692 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] compactions.Compactor(207): Compacting 3bf9532a8609455787c711a66d3557a5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=993, earliestPutTs=1682416596835 2023-04-25 09:56:37,692 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 9f3e98239667445e9b63472e3ae31e36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=993, earliestPutTs=1682416596835 2023-04-25 09:56:37,701 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#642 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:37,715 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#A#compaction#643 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:37,724 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042535e12236ceb74964aca00913a6167f9e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042535e12236ceb74964aca00913a6167f9e 2023-04-25 09:56:37,742 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1029, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/edfae2df963e4909a8ac7610c5e7a5dc 2023-04-25 09:56:37,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:37,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:37,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416657976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416657977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416657977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:37,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:37,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416657979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416658016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416658080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416658080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416658080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416658083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,113 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/ee304d9abedf48869ea2ba2823040965 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/ee304d9abedf48869ea2ba2823040965 2023-04-25 09:56:38,119 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into ee304d9abedf48869ea2ba2823040965(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:38,119 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:38,119 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=13, startTime=1682416597689; duration=0sec 2023-04-25 09:56:38,119 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:38,119 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:38,120 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:38,120 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:38,120 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,120 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/7bdf5efe68c745fdaa05abc5a561cb7c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fae9049c28704c2899b65e956aaa9ad4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a4cced7c680a45988c243889b80c5e5a] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.6 K 2023-04-25 09:56:38,121 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 7bdf5efe68c745fdaa05abc5a561cb7c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=965, earliestPutTs=1682416595675 2023-04-25 09:56:38,121 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting fae9049c28704c2899b65e956aaa9ad4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1682416595691 2023-04-25 09:56:38,121 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting a4cced7c680a45988c243889b80c5e5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=993, earliestPutTs=1682416596835 2023-04-25 09:56:38,128 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#646 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:38,151 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/d1fbb66e0bd14029ab049e75adb8f438 2023-04-25 09:56:38,160 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1029 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/574152c449d7425d9c5a2593bfaa9a96 2023-04-25 09:56:38,163 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/edfae2df963e4909a8ac7610c5e7a5dc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/edfae2df963e4909a8ac7610c5e7a5dc 2023-04-25 09:56:38,167 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/edfae2df963e4909a8ac7610c5e7a5dc, entries=150, sequenceid=1029, filesize=25.4 K 2023-04-25 09:56:38,168 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/d1fbb66e0bd14029ab049e75adb8f438 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d1fbb66e0bd14029ab049e75adb8f438 2023-04-25 09:56:38,171 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d1fbb66e0bd14029ab049e75adb8f438, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-25 09:56:38,171 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/574152c449d7425d9c5a2593bfaa9a96 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/574152c449d7425d9c5a2593bfaa9a96 2023-04-25 09:56:38,175 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/574152c449d7425d9c5a2593bfaa9a96, entries=150, sequenceid=1029, filesize=11.9 K 2023-04-25 09:56:38,175 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for fb513dc5b3a62f2dba9b88a4e6de5394 in 485ms, sequenceid=1029, compaction requested=false 2023-04-25 09:56:38,175 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:38,176 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:38,177 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,177 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea387b8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:38,179 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:38,179 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,180 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:38,181 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:38,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,181 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:38,183 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:38,186 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:38,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:38,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:38,186 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:38,186 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:38,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,187 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,188 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,189 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,191 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:38,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:38,192 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:38,192 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:38,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,191 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,192 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,192 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,192 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:38,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:38,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-25 09:56:38,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:38,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:38,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:38,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:38,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:38,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:38,292 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:38,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:38,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:38,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:38,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:38,297 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:38,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:38,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:38,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,297 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,299 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,299 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,299 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,299 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:38,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:38,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:38,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:38,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:38,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:38,304 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:38,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:38,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:38,305 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:38,305 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:38,309 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:38,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e57157[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:38,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:38,311 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,312 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:38,312 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,312 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:38,313 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,313 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:38,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:38,315 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:38,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:38,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:38,315 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:38,315 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:38,322 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,322 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,322 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,323 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,324 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,325 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,325 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,326 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:38,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:38,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,326 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,326 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:38,326 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:38,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416658332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416658332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416658333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416658333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,426 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:38,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:38,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:38,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:38,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:38,429 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:38,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:38,429 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:38,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,430 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,431 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,431 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:38,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:38,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:38,433 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,433 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:38,433 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:38,433 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:38,434 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,434 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7b6a7f[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:38,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:38,436 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,438 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:38,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:38,439 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,439 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:38,439 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:38,440 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,440 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:38,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416658438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416658438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416658438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416658438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:38,443 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:38,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:38,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:38,443 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:38,443 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:38,444 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,444 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,444 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,445 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,446 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,448 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:38,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:38,448 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,449 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,449 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,449 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:38,449 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,449 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:38,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,539 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/55c6a518423f4ab8b2df959020909651 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/55c6a518423f4ab8b2df959020909651 2023-04-25 09:56:38,544 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into 55c6a518423f4ab8b2df959020909651(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:38,544 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:38,544 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=13, startTime=1682416597689; duration=0sec 2023-04-25 09:56:38,544 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:38,549 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:38,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:38,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:38,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:38,552 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:38,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:38,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:38,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,552 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,554 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,554 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:38,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:38,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:38,556 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,556 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:38,557 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:38,557 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:38,557 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:38,558 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,558 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,559 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:38,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:38,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244dbee4[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:38,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:38,560 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,561 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,562 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:38,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:38,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:38,562 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:38,563 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:38,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:38,567 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:38,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:38,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:38,567 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:38,567 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:38,574 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,574 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,574 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,574 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,575 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,577 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,578 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:38,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:38,578 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,578 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:38,578 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416658642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416658642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416658643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416658643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,679 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:38,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:38,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:38,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:38,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:38,682 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:38,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:38,682 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:38,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,683 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,684 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,684 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:38,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:38,685 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:38,686 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,686 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:38,686 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:38,686 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:38,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:38,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:38,686 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:38,687 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,688 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a17392[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:38,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:38,689 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,696 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:38,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:38,696 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:38,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,696 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:38,699 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,699 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,700 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,700 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:38,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:38,702 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:38,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:38,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:38,702 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:38,702 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:38,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,703 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:38,704 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:38,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:38,705 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:38,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,708 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:38,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:38,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:38,709 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:38,709 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:38,708 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:38,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:38,710 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304258a7db1718a794382bc20f91a8e188488 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258a7db1718a794382bc20f91a8e188488 2023-04-25 09:56:38,715 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1044, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/ca19841d6c124d50bbc66f477cba2333 2023-04-25 09:56:38,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1044 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/d733ffe788c3409d968264368a5264fc 2023-04-25 09:56:38,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1044 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/3e4d9e6ff3754b98be20d9004fe581e4 2023-04-25 09:56:38,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/ca19841d6c124d50bbc66f477cba2333 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ca19841d6c124d50bbc66f477cba2333 2023-04-25 09:56:38,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ca19841d6c124d50bbc66f477cba2333, entries=200, sequenceid=1044, filesize=32.3 K 2023-04-25 09:56:38,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/d733ffe788c3409d968264368a5264fc as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d733ffe788c3409d968264368a5264fc 2023-04-25 09:56:38,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d733ffe788c3409d968264368a5264fc, entries=150, sequenceid=1044, filesize=11.9 K 2023-04-25 09:56:38,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/3e4d9e6ff3754b98be20d9004fe581e4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/3e4d9e6ff3754b98be20d9004fe581e4 2023-04-25 09:56:38,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/3e4d9e6ff3754b98be20d9004fe581e4, entries=150, sequenceid=1044, filesize=11.9 K 2023-04-25 09:56:38,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for fb513dc5b3a62f2dba9b88a4e6de5394 in 455ms, sequenceid=1044, compaction requested=true 2023-04-25 09:56:38,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:38,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:38,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:38,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-25 09:56:38,744 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-25 09:56:38,745 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-25 09:56:38,745 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-25 09:56:38,745 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. because compaction request was cancelled 2023-04-25 09:56:38,745 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:38,746 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:38,746 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/B is initiating minor compaction (all files) 2023-04-25 09:56:38,746 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/B in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,746 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/ee304d9abedf48869ea2ba2823040965, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d1fbb66e0bd14029ab049e75adb8f438, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d733ffe788c3409d968264368a5264fc] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.7 K 2023-04-25 09:56:38,746 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting ee304d9abedf48869ea2ba2823040965, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=993, earliestPutTs=1682416596835 2023-04-25 09:56:38,747 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d1fbb66e0bd14029ab049e75adb8f438, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1029, earliestPutTs=1682416596883 2023-04-25 09:56:38,747 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting d733ffe788c3409d968264368a5264fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1044, earliestPutTs=1682416597975 2023-04-25 09:56:38,751 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#B#compaction#651 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:38,759 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/0bc85551209143ebb79fcc197d19e99e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0bc85551209143ebb79fcc197d19e99e 2023-04-25 09:56:38,763 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/B of fb513dc5b3a62f2dba9b88a4e6de5394 into 0bc85551209143ebb79fcc197d19e99e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:38,764 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:38,764 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/B, priority=13, startTime=1682416598744; duration=0sec 2023-04-25 09:56:38,764 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:38,764 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-25 09:56:38,764 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-25 09:56:38,765 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1912): fb513dc5b3a62f2dba9b88a4e6de5394/C is initiating minor compaction (all files) 2023-04-25 09:56:38,765 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fb513dc5b3a62f2dba9b88a4e6de5394/C in TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,765 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/55c6a518423f4ab8b2df959020909651, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/574152c449d7425d9c5a2593bfaa9a96, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/3e4d9e6ff3754b98be20d9004fe581e4] into tmpdir=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp, totalSize=36.7 K 2023-04-25 09:56:38,766 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 55c6a518423f4ab8b2df959020909651, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=993, earliestPutTs=1682416596835 2023-04-25 09:56:38,766 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 574152c449d7425d9c5a2593bfaa9a96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1029, earliestPutTs=1682416596883 2023-04-25 09:56:38,766 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] compactions.Compactor(207): Compacting 3e4d9e6ff3754b98be20d9004fe581e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1044, earliestPutTs=1682416597975 2023-04-25 09:56:38,771 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] throttle.PressureAwareThroughputController(145): fb513dc5b3a62f2dba9b88a4e6de5394#C#compaction#652 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 61.54 MB/second 2023-04-25 09:56:38,809 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:38,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:38,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-25 09:56:38,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-25 09:56:38,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-25 09:56:38,813 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-25 09:56:38,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-25 09:56:38,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-25 09:56:38,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,813 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,815 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:38,815 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:38,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,816 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-25 09:56:38,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-25 09:56:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:38,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-25 09:56:38,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-25 09:56:38,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-25 09:56:38,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-25 09:56:38,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-25 09:56:38,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-25 09:56:38,817 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-25 09:56:38,818 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. started... 2023-04-25 09:56:38,818 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:38,818 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:38,865 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.15384615384615385, tune throughput to 57.69 MB/second 2023-04-25 09:56:38,948 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/e2bcb1ede51c4d79be0dc114cb24e7eb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e2bcb1ede51c4d79be0dc114cb24e7eb 2023-04-25 09:56:38,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:38,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. as already flushing 2023-04-25 09:56:38,953 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/A of fb513dc5b3a62f2dba9b88a4e6de5394 into e2bcb1ede51c4d79be0dc114cb24e7eb(size=26.4 K), total size for store is 84.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-25 09:56:38,953 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:38,953 INFO [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/A, priority=13, startTime=1682416597689; duration=1sec 2023-04-25 09:56:38,953 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:38,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416658953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416658954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416658955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:38,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:38,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416658957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48606 deadline: 1682416659028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416659057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416659058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416659058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416659060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,180 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/68f3efed425a403e81a7534cb9d8a070 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/68f3efed425a403e81a7534cb9d8a070 2023-04-25 09:56:39,184 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fb513dc5b3a62f2dba9b88a4e6de5394/C of fb513dc5b3a62f2dba9b88a4e6de5394 into 68f3efed425a403e81a7534cb9d8a070(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-25 09:56:39,184 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:39,184 INFO [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394., storeName=fb513dc5b3a62f2dba9b88a4e6de5394/C, priority=13, startTime=1682416598744; duration=0sec 2023-04-25 09:56:39,184 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:39,233 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425783eff7245f641949fee0959a3874c23 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425783eff7245f641949fee0959a3874c23 2023-04-25 09:56:39,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416659261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416659261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416659262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416659264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39407] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416659565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416659565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39407] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416659566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:39,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416659570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:39,638 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1065, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/49c4b15ffd534c4c9ed032e16d941fbb 2023-04-25 09:56:39,769 DEBUG [Thread-4120] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:60014 2023-04-25 09:56:39,769 DEBUG [Thread-4120] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:39,769 DEBUG [Thread-4126] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:60014 2023-04-25 09:56:39,769 DEBUG [Thread-4126] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:39,770 DEBUG [Thread-4124] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:60014 2023-04-25 09:56:39,770 DEBUG [Thread-4124] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:39,772 DEBUG [Thread-4122] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:60014 2023-04-25 09:56:39,772 DEBUG [Thread-4122] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:39,773 DEBUG [Thread-4128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:60014 2023-04-25 09:56:39,773 DEBUG [Thread-4128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:40,047 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1065 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/566a4ba0898c4267b541aff96c84ced5 2023-04-25 09:56:40,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:40,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48690 deadline: 1682416660068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:40,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48616 deadline: 1682416660071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:40,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48638 deadline: 1682416660071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-25 09:56:40,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48692 deadline: 1682416660074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,455 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1065 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/b16fc6d52de2478a9f074307aa04c4b4 2023-04-25 09:56:40,461 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/49c4b15ffd534c4c9ed032e16d941fbb as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49c4b15ffd534c4c9ed032e16d941fbb 2023-04-25 09:56:40,464 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49c4b15ffd534c4c9ed032e16d941fbb, entries=150, sequenceid=1065, filesize=25.4 K 2023-04-25 09:56:40,464 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/566a4ba0898c4267b541aff96c84ced5 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/566a4ba0898c4267b541aff96c84ced5 2023-04-25 09:56:40,467 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/566a4ba0898c4267b541aff96c84ced5, entries=150, sequenceid=1065, filesize=11.9 K 2023-04-25 09:56:40,467 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/b16fc6d52de2478a9f074307aa04c4b4 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b16fc6d52de2478a9f074307aa04c4b4 2023-04-25 09:56:40,470 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b16fc6d52de2478a9f074307aa04c4b4, entries=150, sequenceid=1065, filesize=11.9 K 2023-04-25 09:56:40,470 INFO [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for fb513dc5b3a62f2dba9b88a4e6de5394 in 1652ms, sequenceid=1065, compaction requested=true 2023-04-25 09:56:40,470 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:40,471 DEBUG [rs(jenkins-hbase4.apache.org,39407,1682416417123)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:40,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-25 09:56:40,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-25 09:56:40,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-25 09:56:40,471 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-25 09:56:40,472 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,472 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:40,472 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,473 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-25 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:40,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,39407,1682416417123' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-25 09:56:40,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@890cfd8[Count = 0] remaining members to acquire global barrier 2023-04-25 09:56:40,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-25 09:56:40,474 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,475 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-25 09:56:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-25 09:56:40,475 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,39407,1682416417123' in zk 2023-04-25 09:56:40,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,475 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-25 09:56:40,476 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-25 09:56:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:40,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:40,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,39407,1682416417123' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-25 09:56:40,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,39407,1682416417123': 2023-04-25 09:56:40,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,39407,1682416417123' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-25 09:56:40,478 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-25 09:56:40,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-25 09:56:40,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-25 09:56:40,478 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-25 09:56:40,478 INFO [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-25 09:56:40,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,480 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-25 09:56:40,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,481 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-25 09:56:40,481 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-25 09:56:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:40,482 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-25 09:56:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-25 09:56:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-25 09:56:40,483 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:40,484 DEBUG [(jenkins-hbase4.apache.org,33039,1682416415885)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-25 09:56:40,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-25 09:56:40,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-25 09:56:40,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-25 09:56:40,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-25 09:56:40,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-25 09:56:40,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-25 09:56:40,484 DEBUG [Thread-4119] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-25 09:56:40,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:40,484 DEBUG [Thread-4119] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-25 09:56:40,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-25 09:56:40,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-25 09:56:40,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-25 09:56:40,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-25 09:56:40,484 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-25 09:56:40,584 DEBUG [Thread-4119] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-25 09:56:40,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-25 09:56:41,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39407] regionserver.HRegion(9158): Flush requested on fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:41,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-25 09:56:41,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:41,044 DEBUG [Thread-4111] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:60014 2023-04-25 09:56:41,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:41,044 DEBUG [Thread-4111] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:41,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:41,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:41,045 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:41,045 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:41,073 DEBUG [Thread-4117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:60014 2023-04-25 09:56:41,073 DEBUG [Thread-4117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:41,076 DEBUG [Thread-4109] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:60014 2023-04-25 09:56:41,076 DEBUG [Thread-4109] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:41,078 DEBUG [Thread-4115] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:60014 2023-04-25 09:56:41,078 DEBUG [Thread-4115] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:41,081 DEBUG [Thread-4113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:60014 2023-04-25 09:56:41,081 DEBUG [Thread-4113] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 271 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 284 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-04-25 09:56:41,081 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4579 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4640 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4535 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4586 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(398): read 4670 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:56:41,082 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x470742aa to 127.0.0.1:60014 2023-04-25 09:56:41,082 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:41,082 INFO [Listener at localhost/33705] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-25 09:56:41,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-25 09:56:41,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:41,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:41,086 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416601086"}]},"ts":"1682416601086"} 2023-04-25 09:56:41,086 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-25 09:56:41,094 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-25 09:56:41,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, UNASSIGN}] 2023-04-25 09:56:41,096 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, UNASSIGN 2023-04-25 09:56:41,096 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:41,096 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416601096"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682416601096"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682416601096"}]},"ts":"1682416601096"} 2023-04-25 09:56:41,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123}] 2023-04-25 09:56:41,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:41,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:41,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fb513dc5b3a62f2dba9b88a4e6de5394, disabling compactions & flushes 2023-04-25 09:56:41,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:41,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:41,455 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425137ab2a3ffb14ea6aaeeab5366888ade to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425137ab2a3ffb14ea6aaeeab5366888ade 2023-04-25 09:56:41,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:41,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1081, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/7d7cfee7bd074d2b94558d3e0a7ac88b 2023-04-25 09:56:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:42,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/f3fdfed5e77443bb96e491d28756e790 2023-04-25 09:56:42,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/173b1bb7b55540d4bfa4a13859c831a6 2023-04-25 09:56:42,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/7d7cfee7bd074d2b94558d3e0a7ac88b as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/7d7cfee7bd074d2b94558d3e0a7ac88b 2023-04-25 09:56:42,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/7d7cfee7bd074d2b94558d3e0a7ac88b, entries=150, sequenceid=1081, filesize=25.4 K 2023-04-25 09:56:42,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/f3fdfed5e77443bb96e491d28756e790 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f3fdfed5e77443bb96e491d28756e790 2023-04-25 09:56:42,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f3fdfed5e77443bb96e491d28756e790, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-25 09:56:42,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/173b1bb7b55540d4bfa4a13859c831a6 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/173b1bb7b55540d4bfa4a13859c831a6 2023-04-25 09:56:42,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/173b1bb7b55540d4bfa4a13859c831a6, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-25 09:56:42,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=26.84 KB/27480 for fb513dc5b3a62f2dba9b88a4e6de5394 in 1646ms, sequenceid=1081, compaction requested=true 2023-04-25 09:56:42,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:42,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:42,690 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. because compaction request was cancelled 2023-04-25 09:56:42,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:42,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. after waiting 0 ms 2023-04-25 09:56:42,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:42,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:42,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-25 09:56:42,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fb513dc5b3a62f2dba9b88a4e6de5394 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-25 09:56:42,690 DEBUG [RS:0;jenkins-hbase4:39407-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. because compaction request was cancelled 2023-04-25 09:56:42,690 DEBUG [RS:0;jenkins-hbase4:39407-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. because compaction request was cancelled 2023-04-25 09:56:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=A 2023-04-25 09:56:42,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-25 09:56:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=B 2023-04-25 09:56:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fb513dc5b3a62f2dba9b88a4e6de5394, store=C 2023-04-25 09:56:42,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-25 09:56:43,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230425edca79c2e9bb49eb916e5a4c178cdd3c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425edca79c2e9bb49eb916e5a4c178cdd3c 2023-04-25 09:56:43,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:43,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1088, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/c2837a3cadc4485fac539635269fc7d0 2023-04-25 09:56:43,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1088 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/7e1afaed58b3450da8bfda333de18e3c 2023-04-25 09:56:44,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1088 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/fd6c05198143432bad398adbb1dfe43a 2023-04-25 09:56:44,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/A/c2837a3cadc4485fac539635269fc7d0 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c2837a3cadc4485fac539635269fc7d0 2023-04-25 09:56:44,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c2837a3cadc4485fac539635269fc7d0, entries=150, sequenceid=1088, filesize=25.4 K 2023-04-25 09:56:44,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/B/7e1afaed58b3450da8bfda333de18e3c as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7e1afaed58b3450da8bfda333de18e3c 2023-04-25 09:56:44,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7e1afaed58b3450da8bfda333de18e3c, entries=150, sequenceid=1088, filesize=11.9 K 2023-04-25 09:56:44,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/.tmp/C/fd6c05198143432bad398adbb1dfe43a as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fd6c05198143432bad398adbb1dfe43a 2023-04-25 09:56:44,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fd6c05198143432bad398adbb1dfe43a, entries=150, sequenceid=1088, filesize=11.9 K 2023-04-25 09:56:44,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for fb513dc5b3a62f2dba9b88a4e6de5394 in 1651ms, sequenceid=1088, compaction requested=true 2023-04-25 09:56:44,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/cc66ba0e57c74b1a9edcb9aeb87d7662, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d35ef76f9384648b35497c4c33744ea, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c23bd2f6f7884b868189ca5c6696ffd3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f745bfa0182b46609026b06117303013, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9ac3cac0113541b382a0877736bde983, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9a89a8a622d4451497819bc8c614b886, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f0d861aa241c436f9ce1b6d31cee52a5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/eef0298bc39247cc83bccdfe6acf237b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a0ff9369e3754f449100957369b08199, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/17f0fa17bbfc4eadb81e355ed097048f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c3cc01e44a8746be9de6380174394010, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49946d52b48b498f81390e14283f3410, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/947d80a73db3471d8d15d05c9522a8ae, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/417cd5404540469591f690d4e071da97, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d9a9be6a24a4c0393705ab5be890897, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/579a016c964c4a27a4e14fcee23f2473, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a68bc2680f5f46f1b20c03524cab53b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/df7969ab47b34195bed32ac6ac2c1d9b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/519a6b4d3b904f9e852c776c86c767ed, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ddfa75d380e5449394d375f35e93e7f2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/0ec72ae8864a4e4d884bac814ef7ca78, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/819cd86bb0934c3f802dc0848feac982, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/023f3e83ab5341359b2a54425e4c5ddc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/1f3c873876574bc0b277061c4ba1feba, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/bb193bbbd5d04672949d36c1e52e51b6, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e3a19034c96d47c9888913fdecd9e0dc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/660b13f8c2004c51acdda1c823cf68d7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/4b802a1af5834b27b40aefb4f22b754b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/64d80a3805ee496ab49008c2c354c341, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/3bf9532a8609455787c711a66d3557a5] to archive 2023-04-25 09:56:44,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:56:44,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/cc66ba0e57c74b1a9edcb9aeb87d7662 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/cc66ba0e57c74b1a9edcb9aeb87d7662 2023-04-25 09:56:44,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d35ef76f9384648b35497c4c33744ea to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d35ef76f9384648b35497c4c33744ea 2023-04-25 09:56:44,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c23bd2f6f7884b868189ca5c6696ffd3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c23bd2f6f7884b868189ca5c6696ffd3 2023-04-25 09:56:44,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f745bfa0182b46609026b06117303013 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f745bfa0182b46609026b06117303013 2023-04-25 09:56:44,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9ac3cac0113541b382a0877736bde983 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9ac3cac0113541b382a0877736bde983 2023-04-25 09:56:44,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9a89a8a622d4451497819bc8c614b886 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/9a89a8a622d4451497819bc8c614b886 2023-04-25 09:56:44,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f0d861aa241c436f9ce1b6d31cee52a5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/f0d861aa241c436f9ce1b6d31cee52a5 2023-04-25 09:56:44,356 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/eef0298bc39247cc83bccdfe6acf237b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/eef0298bc39247cc83bccdfe6acf237b 2023-04-25 09:56:44,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a0ff9369e3754f449100957369b08199 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a0ff9369e3754f449100957369b08199 2023-04-25 09:56:44,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/17f0fa17bbfc4eadb81e355ed097048f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/17f0fa17bbfc4eadb81e355ed097048f 2023-04-25 09:56:44,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c3cc01e44a8746be9de6380174394010 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c3cc01e44a8746be9de6380174394010 2023-04-25 09:56:44,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49946d52b48b498f81390e14283f3410 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49946d52b48b498f81390e14283f3410 2023-04-25 09:56:44,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/947d80a73db3471d8d15d05c9522a8ae to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/947d80a73db3471d8d15d05c9522a8ae 2023-04-25 09:56:44,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/417cd5404540469591f690d4e071da97 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/417cd5404540469591f690d4e071da97 2023-04-25 09:56:44,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d9a9be6a24a4c0393705ab5be890897 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/8d9a9be6a24a4c0393705ab5be890897 2023-04-25 09:56:44,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/579a016c964c4a27a4e14fcee23f2473 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/579a016c964c4a27a4e14fcee23f2473 2023-04-25 09:56:44,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a68bc2680f5f46f1b20c03524cab53b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/a68bc2680f5f46f1b20c03524cab53b1 2023-04-25 09:56:44,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/df7969ab47b34195bed32ac6ac2c1d9b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/df7969ab47b34195bed32ac6ac2c1d9b 2023-04-25 09:56:44,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/519a6b4d3b904f9e852c776c86c767ed to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/519a6b4d3b904f9e852c776c86c767ed 2023-04-25 09:56:44,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ddfa75d380e5449394d375f35e93e7f2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ddfa75d380e5449394d375f35e93e7f2 2023-04-25 09:56:44,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/0ec72ae8864a4e4d884bac814ef7ca78 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/0ec72ae8864a4e4d884bac814ef7ca78 2023-04-25 09:56:44,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/819cd86bb0934c3f802dc0848feac982 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/819cd86bb0934c3f802dc0848feac982 2023-04-25 09:56:44,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/023f3e83ab5341359b2a54425e4c5ddc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/023f3e83ab5341359b2a54425e4c5ddc 2023-04-25 09:56:44,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/1f3c873876574bc0b277061c4ba1feba to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/1f3c873876574bc0b277061c4ba1feba 2023-04-25 09:56:44,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/bb193bbbd5d04672949d36c1e52e51b6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/bb193bbbd5d04672949d36c1e52e51b6 2023-04-25 09:56:44,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e3a19034c96d47c9888913fdecd9e0dc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e3a19034c96d47c9888913fdecd9e0dc 2023-04-25 09:56:44,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/660b13f8c2004c51acdda1c823cf68d7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/660b13f8c2004c51acdda1c823cf68d7 2023-04-25 09:56:44,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/4b802a1af5834b27b40aefb4f22b754b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/4b802a1af5834b27b40aefb4f22b754b 2023-04-25 09:56:44,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/64d80a3805ee496ab49008c2c354c341 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/64d80a3805ee496ab49008c2c354c341 2023-04-25 09:56:44,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/3bf9532a8609455787c711a66d3557a5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/3bf9532a8609455787c711a66d3557a5 2023-04-25 09:56:44,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/8266e7f1099447cb86f6357d1925bf68, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2def4ad8c24b455f86bbe14a0f425af3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/48ec436bdfb4499eaf943afbdb770930, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/10bbf78705d248a48f6c003670e82edc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6c684ea3c8484d9885d0ee9bf511d101, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2207490c04a842ca9c0c0da88077d64f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/de0390119bee4e6db6916d09c6c6bbab, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/75695efcd2334ba78f121fd20b020514, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/928d3e6ce1564e8c98b3244ba8539e8f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/838e555a13e8406a95b9cb2d21db5ddd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/44eb7a8175114623a4d38c418dbe6e36, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0d89b868835744c9876993c5f0c2d3b4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2fe9e2889c13467c9023bdfcb8c580b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6f2a1f16ac5c4e33895addfe4dce39bd, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/3067cce0113243d9b1f79859d5d39066, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/e7bd06f7a8f8458ca9dd5c78dc55fa50, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/299845aabe7e4304969bb5fbdeedad30, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0cfe0fc552d548288c56e67c556bfd18, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a1d15e9e437b4fab86cfb4b17e2c8334, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/90a9250f2bbd4337a968e81456954e90, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/41017b0fdaa043c695ee5352679d8d51, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/985f1e34a251451eb5e2bd085b36b169, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/24986282551f459f9bbbc7233b16a5e8, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f5db3a865fa54eb6920b75aa31807d98, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7528c01a456642d38155dbb579fe8b99, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/1c56b6be557c4c84a28297199295fdb2, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/532b01cd9e9f415b83352ad9339f882f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a8042e3a45bb4376bd284d204b4236db, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/261ede963f134021b489055483e3357a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/ee304d9abedf48869ea2ba2823040965, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/9f3e98239667445e9b63472e3ae31e36, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d1fbb66e0bd14029ab049e75adb8f438, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d733ffe788c3409d968264368a5264fc] to archive 2023-04-25 09:56:44,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:56:44,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/8266e7f1099447cb86f6357d1925bf68 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/8266e7f1099447cb86f6357d1925bf68 2023-04-25 09:56:44,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2def4ad8c24b455f86bbe14a0f425af3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2def4ad8c24b455f86bbe14a0f425af3 2023-04-25 09:56:44,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/48ec436bdfb4499eaf943afbdb770930 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/48ec436bdfb4499eaf943afbdb770930 2023-04-25 09:56:44,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/10bbf78705d248a48f6c003670e82edc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/10bbf78705d248a48f6c003670e82edc 2023-04-25 09:56:44,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6c684ea3c8484d9885d0ee9bf511d101 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6c684ea3c8484d9885d0ee9bf511d101 2023-04-25 09:56:44,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2207490c04a842ca9c0c0da88077d64f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2207490c04a842ca9c0c0da88077d64f 2023-04-25 09:56:44,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/de0390119bee4e6db6916d09c6c6bbab to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/de0390119bee4e6db6916d09c6c6bbab 2023-04-25 09:56:44,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/75695efcd2334ba78f121fd20b020514 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/75695efcd2334ba78f121fd20b020514 2023-04-25 09:56:44,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/928d3e6ce1564e8c98b3244ba8539e8f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/928d3e6ce1564e8c98b3244ba8539e8f 2023-04-25 09:56:44,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/838e555a13e8406a95b9cb2d21db5ddd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/838e555a13e8406a95b9cb2d21db5ddd 2023-04-25 09:56:44,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/44eb7a8175114623a4d38c418dbe6e36 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/44eb7a8175114623a4d38c418dbe6e36 2023-04-25 09:56:44,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0d89b868835744c9876993c5f0c2d3b4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0d89b868835744c9876993c5f0c2d3b4 2023-04-25 09:56:44,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2fe9e2889c13467c9023bdfcb8c580b9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/2fe9e2889c13467c9023bdfcb8c580b9 2023-04-25 09:56:44,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6f2a1f16ac5c4e33895addfe4dce39bd to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/6f2a1f16ac5c4e33895addfe4dce39bd 2023-04-25 09:56:44,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/3067cce0113243d9b1f79859d5d39066 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/3067cce0113243d9b1f79859d5d39066 2023-04-25 09:56:44,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/e7bd06f7a8f8458ca9dd5c78dc55fa50 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/e7bd06f7a8f8458ca9dd5c78dc55fa50 2023-04-25 09:56:44,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/299845aabe7e4304969bb5fbdeedad30 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/299845aabe7e4304969bb5fbdeedad30 2023-04-25 09:56:44,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0cfe0fc552d548288c56e67c556bfd18 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0cfe0fc552d548288c56e67c556bfd18 2023-04-25 09:56:44,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a1d15e9e437b4fab86cfb4b17e2c8334 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a1d15e9e437b4fab86cfb4b17e2c8334 2023-04-25 09:56:44,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/90a9250f2bbd4337a968e81456954e90 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/90a9250f2bbd4337a968e81456954e90 2023-04-25 09:56:44,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/41017b0fdaa043c695ee5352679d8d51 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/41017b0fdaa043c695ee5352679d8d51 2023-04-25 09:56:44,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/985f1e34a251451eb5e2bd085b36b169 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/985f1e34a251451eb5e2bd085b36b169 2023-04-25 09:56:44,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/24986282551f459f9bbbc7233b16a5e8 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/24986282551f459f9bbbc7233b16a5e8 2023-04-25 09:56:44,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f5db3a865fa54eb6920b75aa31807d98 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f5db3a865fa54eb6920b75aa31807d98 2023-04-25 09:56:44,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7528c01a456642d38155dbb579fe8b99 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7528c01a456642d38155dbb579fe8b99 2023-04-25 09:56:44,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/1c56b6be557c4c84a28297199295fdb2 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/1c56b6be557c4c84a28297199295fdb2 2023-04-25 09:56:44,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/532b01cd9e9f415b83352ad9339f882f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/532b01cd9e9f415b83352ad9339f882f 2023-04-25 09:56:44,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a8042e3a45bb4376bd284d204b4236db to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/a8042e3a45bb4376bd284d204b4236db 2023-04-25 09:56:44,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/261ede963f134021b489055483e3357a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/261ede963f134021b489055483e3357a 2023-04-25 09:56:44,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/ee304d9abedf48869ea2ba2823040965 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/ee304d9abedf48869ea2ba2823040965 2023-04-25 09:56:44,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/9f3e98239667445e9b63472e3ae31e36 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/9f3e98239667445e9b63472e3ae31e36 2023-04-25 09:56:44,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d1fbb66e0bd14029ab049e75adb8f438 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d1fbb66e0bd14029ab049e75adb8f438 2023-04-25 09:56:44,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d733ffe788c3409d968264368a5264fc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/d733ffe788c3409d968264368a5264fc 2023-04-25 09:56:44,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4b2933c90f394289bbc58b1686374ee9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5c5ce91e76df4d4fae119d550adb50fc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f79a2495edc246afbdbd58cbba30dc25, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/0ee8bf3e86cd4ac58e42d65d484dbf0d, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4be2f078448f41e3a762b75466e835e5, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e028db4d775244bb832188681eea166b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/abead543676344ab956db99267ad3e4e, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/6a99ba85684345f98f3da1077ffa262b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/91cdda41fa5f43b48a6746a5400a233c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/af09e97564b14bb1aebe31bd6e8764ad, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/ddeba9b3dc8a418e80906db2afd7f3b9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/9516e23f55d5469e94c7641eb03fe9b3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/199d0f6c6d0a42df824f6a456fd48b2b, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/2ebae39ad66f4bda92c51ba516010377, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5e9fbf1c80324b67bbfe66f4a742b70f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b882a59003014c10b541f821ff369253, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a191433db0984f0fa7696ef3e31f22b1, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/afaca8477b87428e92b9a224c8e8b358, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/d8e90193707e4845847fcd01c1fec008, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/bf133d1362f546cea4fa8fd923af39da, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/78fccc927b7e4b28b2d95184b148aafc, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/804c304d15f3464d85bf61336f2ffbb7, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/84eb1343f4374c8f92cfa4c55cfe4e2f, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e14c34298d4b4ac696eece5892df6cf9, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/02eb04aed04343aca3de204c15d96c56, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e7f7b2ccca764ca8b72f35ed307cd975, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/7bdf5efe68c745fdaa05abc5a561cb7c, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f7fab86679b546e9bb8c7c0957174af3, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fae9049c28704c2899b65e956aaa9ad4, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/55c6a518423f4ab8b2df959020909651, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a4cced7c680a45988c243889b80c5e5a, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/574152c449d7425d9c5a2593bfaa9a96, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/3e4d9e6ff3754b98be20d9004fe581e4] to archive 2023-04-25 09:56:44,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-25 09:56:44,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4b2933c90f394289bbc58b1686374ee9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4b2933c90f394289bbc58b1686374ee9 2023-04-25 09:56:44,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5c5ce91e76df4d4fae119d550adb50fc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5c5ce91e76df4d4fae119d550adb50fc 2023-04-25 09:56:44,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f79a2495edc246afbdbd58cbba30dc25 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f79a2495edc246afbdbd58cbba30dc25 2023-04-25 09:56:44,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/0ee8bf3e86cd4ac58e42d65d484dbf0d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/0ee8bf3e86cd4ac58e42d65d484dbf0d 2023-04-25 09:56:44,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4be2f078448f41e3a762b75466e835e5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/4be2f078448f41e3a762b75466e835e5 2023-04-25 09:56:44,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e028db4d775244bb832188681eea166b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e028db4d775244bb832188681eea166b 2023-04-25 09:56:44,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/abead543676344ab956db99267ad3e4e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/abead543676344ab956db99267ad3e4e 2023-04-25 09:56:44,415 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/6a99ba85684345f98f3da1077ffa262b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/6a99ba85684345f98f3da1077ffa262b 2023-04-25 09:56:44,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/91cdda41fa5f43b48a6746a5400a233c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/91cdda41fa5f43b48a6746a5400a233c 2023-04-25 09:56:44,417 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/af09e97564b14bb1aebe31bd6e8764ad to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/af09e97564b14bb1aebe31bd6e8764ad 2023-04-25 09:56:44,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/ddeba9b3dc8a418e80906db2afd7f3b9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/ddeba9b3dc8a418e80906db2afd7f3b9 2023-04-25 09:56:44,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/9516e23f55d5469e94c7641eb03fe9b3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/9516e23f55d5469e94c7641eb03fe9b3 2023-04-25 09:56:44,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/199d0f6c6d0a42df824f6a456fd48b2b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/199d0f6c6d0a42df824f6a456fd48b2b 2023-04-25 09:56:44,420 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/2ebae39ad66f4bda92c51ba516010377 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/2ebae39ad66f4bda92c51ba516010377 2023-04-25 09:56:44,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5e9fbf1c80324b67bbfe66f4a742b70f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/5e9fbf1c80324b67bbfe66f4a742b70f 2023-04-25 09:56:44,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b882a59003014c10b541f821ff369253 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b882a59003014c10b541f821ff369253 2023-04-25 09:56:44,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a191433db0984f0fa7696ef3e31f22b1 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a191433db0984f0fa7696ef3e31f22b1 2023-04-25 09:56:44,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/afaca8477b87428e92b9a224c8e8b358 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/afaca8477b87428e92b9a224c8e8b358 2023-04-25 09:56:44,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/d8e90193707e4845847fcd01c1fec008 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/d8e90193707e4845847fcd01c1fec008 2023-04-25 09:56:44,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/bf133d1362f546cea4fa8fd923af39da to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/bf133d1362f546cea4fa8fd923af39da 2023-04-25 09:56:44,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/78fccc927b7e4b28b2d95184b148aafc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/78fccc927b7e4b28b2d95184b148aafc 2023-04-25 09:56:44,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/804c304d15f3464d85bf61336f2ffbb7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/804c304d15f3464d85bf61336f2ffbb7 2023-04-25 09:56:44,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/84eb1343f4374c8f92cfa4c55cfe4e2f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/84eb1343f4374c8f92cfa4c55cfe4e2f 2023-04-25 09:56:44,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e14c34298d4b4ac696eece5892df6cf9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e14c34298d4b4ac696eece5892df6cf9 2023-04-25 09:56:44,428 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/02eb04aed04343aca3de204c15d96c56 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/02eb04aed04343aca3de204c15d96c56 2023-04-25 09:56:44,429 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e7f7b2ccca764ca8b72f35ed307cd975 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/e7f7b2ccca764ca8b72f35ed307cd975 2023-04-25 09:56:44,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/7bdf5efe68c745fdaa05abc5a561cb7c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/7bdf5efe68c745fdaa05abc5a561cb7c 2023-04-25 09:56:44,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f7fab86679b546e9bb8c7c0957174af3 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/f7fab86679b546e9bb8c7c0957174af3 2023-04-25 09:56:44,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fae9049c28704c2899b65e956aaa9ad4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fae9049c28704c2899b65e956aaa9ad4 2023-04-25 09:56:44,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/55c6a518423f4ab8b2df959020909651 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/55c6a518423f4ab8b2df959020909651 2023-04-25 09:56:44,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a4cced7c680a45988c243889b80c5e5a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/a4cced7c680a45988c243889b80c5e5a 2023-04-25 09:56:44,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/574152c449d7425d9c5a2593bfaa9a96 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/574152c449d7425d9c5a2593bfaa9a96 2023-04-25 09:56:44,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/3e4d9e6ff3754b98be20d9004fe581e4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/3e4d9e6ff3754b98be20d9004fe581e4 2023-04-25 09:56:44,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/recovered.edits/1091.seqid, newMaxSeqId=1091, maxSeqId=4 2023-04-25 09:56:44,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394. 2023-04-25 09:56:44,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fb513dc5b3a62f2dba9b88a4e6de5394: 2023-04-25 09:56:44,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:44,440 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=fb513dc5b3a62f2dba9b88a4e6de5394, regionState=CLOSED 2023-04-25 09:56:44,441 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682416604440"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682416604440"}]},"ts":"1682416604440"} 2023-04-25 09:56:44,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-25 09:56:44,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure fb513dc5b3a62f2dba9b88a4e6de5394, server=jenkins-hbase4.apache.org,39407,1682416417123 in 3.3450 sec 2023-04-25 09:56:44,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-25 09:56:44,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fb513dc5b3a62f2dba9b88a4e6de5394, UNASSIGN in 3.3490 sec 2023-04-25 09:56:44,444 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682416604444"}]},"ts":"1682416604444"} 2023-04-25 09:56:44,445 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-25 09:56:44,446 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-25 09:56:44,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.3640 sec 2023-04-25 09:56:45,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-25 09:56:45,191 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-25 09:56:45,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-25 09:56:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:45,193 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:45,194 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:45,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-25 09:56:45,197 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:45,199 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C, FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/recovered.edits] 2023-04-25 09:56:45,202 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49c4b15ffd534c4c9ed032e16d941fbb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/49c4b15ffd534c4c9ed032e16d941fbb 2023-04-25 09:56:45,203 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/7d7cfee7bd074d2b94558d3e0a7ac88b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/7d7cfee7bd074d2b94558d3e0a7ac88b 2023-04-25 09:56:45,204 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c2837a3cadc4485fac539635269fc7d0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/c2837a3cadc4485fac539635269fc7d0 2023-04-25 09:56:45,205 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ca19841d6c124d50bbc66f477cba2333 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/ca19841d6c124d50bbc66f477cba2333 2023-04-25 09:56:45,207 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e2bcb1ede51c4d79be0dc114cb24e7eb to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/e2bcb1ede51c4d79be0dc114cb24e7eb 2023-04-25 09:56:45,208 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/edfae2df963e4909a8ac7610c5e7a5dc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/A/edfae2df963e4909a8ac7610c5e7a5dc 2023-04-25 09:56:45,211 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0bc85551209143ebb79fcc197d19e99e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/0bc85551209143ebb79fcc197d19e99e 2023-04-25 09:56:45,212 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/566a4ba0898c4267b541aff96c84ced5 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/566a4ba0898c4267b541aff96c84ced5 2023-04-25 09:56:45,213 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7e1afaed58b3450da8bfda333de18e3c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/7e1afaed58b3450da8bfda333de18e3c 2023-04-25 09:56:45,215 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f3fdfed5e77443bb96e491d28756e790 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/B/f3fdfed5e77443bb96e491d28756e790 2023-04-25 09:56:45,217 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/173b1bb7b55540d4bfa4a13859c831a6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/173b1bb7b55540d4bfa4a13859c831a6 2023-04-25 09:56:45,218 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/68f3efed425a403e81a7534cb9d8a070 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/68f3efed425a403e81a7534cb9d8a070 2023-04-25 09:56:45,219 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b16fc6d52de2478a9f074307aa04c4b4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/b16fc6d52de2478a9f074307aa04c4b4 2023-04-25 09:56:45,220 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fd6c05198143432bad398adbb1dfe43a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/C/fd6c05198143432bad398adbb1dfe43a 2023-04-25 09:56:45,223 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/recovered.edits/1091.seqid to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394/recovered.edits/1091.seqid 2023-04-25 09:56:45,223 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/.tmp/data/default/TestAcidGuarantees/fb513dc5b3a62f2dba9b88a4e6de5394 2023-04-25 09:56:45,223 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-25 09:56:45,224 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 09:56:45,225 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-25 09:56:45,227 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425091e3f85b2614dc68888a182b890d7c9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425091e3f85b2614dc68888a182b890d7c9 2023-04-25 09:56:45,228 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042512374c55f3d84634aa6ef84ecca58dc0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042512374c55f3d84634aa6ef84ecca58dc0 2023-04-25 09:56:45,229 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425137ab2a3ffb14ea6aaeeab5366888ade to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425137ab2a3ffb14ea6aaeeab5366888ade 2023-04-25 09:56:45,230 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425150928f866c24f1a96a76a38ae478981 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425150928f866c24f1a96a76a38ae478981 2023-04-25 09:56:45,231 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251d8099cad97c4b60a5bb2713ac400a92 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304251d8099cad97c4b60a5bb2713ac400a92 2023-04-25 09:56:45,232 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042523f0e45344d74d3fb57b9fd8312e2c66 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042523f0e45344d74d3fb57b9fd8312e2c66 2023-04-25 09:56:45,233 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252f906ac1af484516a0b3922165504816 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304252f906ac1af484516a0b3922165504816 2023-04-25 09:56:45,234 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042535c7b905e59641cba4040d5d0a6ef871 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042535c7b905e59641cba4040d5d0a6ef871 2023-04-25 09:56:45,235 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042535e12236ceb74964aca00913a6167f9e to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042535e12236ceb74964aca00913a6167f9e 2023-04-25 09:56:45,235 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253c769a3c823f4972bc5f83361cc032e0 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304253c769a3c823f4972bc5f83361cc032e0 2023-04-25 09:56:45,237 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255d7f0f12e2784ab49dd4f9458cfea30b to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255d7f0f12e2784ab49dd4f9458cfea30b 2023-04-25 09:56:45,238 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255e85d0e87a2f4fabb7c3469b50c412a6 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255e85d0e87a2f4fabb7c3469b50c412a6 2023-04-25 09:56:45,239 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255fb1a1e9002f4a34857c4880f74d9207 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304255fb1a1e9002f4a34857c4880f74d9207 2023-04-25 09:56:45,240 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042562f6207c968a4cccaf37921ff73c6a4a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042562f6207c968a4cccaf37921ff73c6a4a 2023-04-25 09:56:45,241 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257305856d111c43cebd5fe140a40c0ba7 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304257305856d111c43cebd5fe140a40c0ba7 2023-04-25 09:56:45,242 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042574e14a453ad24c9eb9e298a98297de0f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042574e14a453ad24c9eb9e298a98297de0f 2023-04-25 09:56:45,243 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425783eff7245f641949fee0959a3874c23 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425783eff7245f641949fee0959a3874c23 2023-04-25 09:56:45,244 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258a7db1718a794382bc20f91a8e188488 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304258a7db1718a794382bc20f91a8e188488 2023-04-25 09:56:45,245 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042590751f080f174b429f2e216309ecda04 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042590751f080f174b429f2e216309ecda04 2023-04-25 09:56:45,246 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425af4b059d0c8c43a5aeb185f5c5490cdc to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425af4b059d0c8c43a5aeb185f5c5490cdc 2023-04-25 09:56:45,247 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bde1f9954ac8416482406d8d07b8051d to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425bde1f9954ac8416482406d8d07b8051d 2023-04-25 09:56:45,248 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c52854df27a447eb8a9a80bb767f8336 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425c52854df27a447eb8a9a80bb767f8336 2023-04-25 09:56:45,249 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cef11b978dc14406878f85d37de9a63a to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425cef11b978dc14406878f85d37de9a63a 2023-04-25 09:56:45,250 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d878255c87ca4c8081c74bcb09b1e049 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d878255c87ca4c8081c74bcb09b1e049 2023-04-25 09:56:45,251 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d9004c584bc94787aa6f60fb832c55a9 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425d9004c584bc94787aa6f60fb832c55a9 2023-04-25 09:56:45,252 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e3092573f90b4bd6adcdadf183d784c4 to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425e3092573f90b4bd6adcdadf183d784c4 2023-04-25 09:56:45,253 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425edca79c2e9bb49eb916e5a4c178cdd3c to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425edca79c2e9bb49eb916e5a4c178cdd3c 2023-04-25 09:56:45,254 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f03a4e4461704cc08e611946096df31f to hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230425f03a4e4461704cc08e611946096df31f 2023-04-25 09:56:45,254 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-25 09:56:45,256 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:45,257 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-25 09:56:45,259 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-25 09:56:45,260 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:45,260 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-25 09:56:45,260 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682416605260"}]},"ts":"9223372036854775807"} 2023-04-25 09:56:45,262 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-25 09:56:45,262 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fb513dc5b3a62f2dba9b88a4e6de5394, NAME => 'TestAcidGuarantees,,1682416576676.fb513dc5b3a62f2dba9b88a4e6de5394.', STARTKEY => '', ENDKEY => ''}] 2023-04-25 09:56:45,262 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-25 09:56:45,262 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682416605262"}]},"ts":"9223372036854775807"} 2023-04-25 09:56:45,263 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-25 09:56:45,264 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-25 09:56:45,265 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-04-25 09:56:45,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33039] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-25 09:56:45,295 INFO [Listener at localhost/33705] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-25 09:56:45,307 INFO [Listener at localhost/33705] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 231), OpenFileDescriptor=576 (was 584), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=429 (was 397) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=4442 (was 4526) 2023-04-25 09:56:45,307 INFO [Listener at localhost/33705] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-25 09:56:45,307 INFO [Listener at localhost/33705] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-25 09:56:45,307 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x373f9c99 to 127.0.0.1:60014 2023-04-25 09:56:45,308 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:45,308 DEBUG [Listener at localhost/33705] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-25 09:56:45,308 DEBUG [Listener at localhost/33705] util.JVMClusterUtil(257): Found active master hash=1143794569, stopped=false 2023-04-25 09:56:45,308 INFO [Listener at localhost/33705] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:56:45,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-25 09:56:45,309 INFO [Listener at localhost/33705] procedure2.ProcedureExecutor(629): Stopping 2023-04-25 09:56:45,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:56:45,309 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-25 09:56:45,310 DEBUG [Listener at localhost/33705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2881fe27 to 127.0.0.1:60014 2023-04-25 09:56:45,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 09:56:45,310 DEBUG [Listener at localhost/33705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:45,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-25 09:56:45,310 INFO [Listener at localhost/33705] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39407,1682416417123' ***** 2023-04-25 09:56:45,310 INFO [Listener at localhost/33705] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] regionserver.HeapMemoryManager(220): Stopping 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-25 09:56:45,311 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(3304): Received CLOSE for 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:45,311 DEBUG [RS:0;jenkins-hbase4:39407] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42bc3cbc to 127.0.0.1:60014 2023-04-25 09:56:45,311 DEBUG [RS:0;jenkins-hbase4:39407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-25 09:56:45,311 INFO [RS:0;jenkins-hbase4:39407] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-25 09:56:45,312 INFO [RS:0;jenkins-hbase4:39407] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-25 09:56:45,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4f0d7b338452e6b7e1ce07e6771dd975, disabling compactions & flushes 2023-04-25 09:56:45,312 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-25 09:56:45,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:56:45,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:56:45,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. after waiting 0 ms 2023-04-25 09:56:45,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:56:45,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4f0d7b338452e6b7e1ce07e6771dd975 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-25 09:56:45,313 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-25 09:56:45,313 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1479): Online Regions={4f0d7b338452e6b7e1ce07e6771dd975=hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975., 1588230740=hbase:meta,,1.1588230740} 2023-04-25 09:56:45,313 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-25 09:56:45,313 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-25 09:56:45,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-25 09:56:45,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-25 09:56:45,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-25 09:56:45,314 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-25 09:56:45,315 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1505): Waiting on 1588230740, 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:56:45,336 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-25 09:56:45,515 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1505): Waiting on 1588230740, 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:56:45,716 DEBUG [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1505): Waiting on 1588230740, 4f0d7b338452e6b7e1ce07e6771dd975 2023-04-25 09:56:45,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/.tmp/info/39a5e4d0e92041cc8f91809fee0a412d 2023-04-25 09:56:45,721 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/.tmp/info/6529229d113841afa45dddddd04dd573 2023-04-25 09:56:45,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/.tmp/info/39a5e4d0e92041cc8f91809fee0a412d as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/info/39a5e4d0e92041cc8f91809fee0a412d 2023-04-25 09:56:45,726 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6529229d113841afa45dddddd04dd573 2023-04-25 09:56:45,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/info/39a5e4d0e92041cc8f91809fee0a412d, entries=2, sequenceid=6, filesize=4.8 K 2023-04-25 09:56:45,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4f0d7b338452e6b7e1ce07e6771dd975 in 417ms, sequenceid=6, compaction requested=false 2023-04-25 09:56:45,732 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/.tmp/rep_barrier/9a266351ea0e4355896e1a747caef6f7 2023-04-25 09:56:45,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/namespace/4f0d7b338452e6b7e1ce07e6771dd975/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-25 09:56:45,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:56:45,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4f0d7b338452e6b7e1ce07e6771dd975: 2023-04-25 09:56:45,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682416419475.4f0d7b338452e6b7e1ce07e6771dd975. 2023-04-25 09:56:45,736 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9a266351ea0e4355896e1a747caef6f7 2023-04-25 09:56:45,740 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/.tmp/table/3bb3be9613c24e3d821d22970a01e34e 2023-04-25 09:56:45,743 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3bb3be9613c24e3d821d22970a01e34e 2023-04-25 09:56:45,743 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/.tmp/info/6529229d113841afa45dddddd04dd573 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/info/6529229d113841afa45dddddd04dd573 2023-04-25 09:56:45,745 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6529229d113841afa45dddddd04dd573 2023-04-25 09:56:45,746 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/info/6529229d113841afa45dddddd04dd573, entries=22, sequenceid=93, filesize=7.1 K 2023-04-25 09:56:45,747 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/.tmp/rep_barrier/9a266351ea0e4355896e1a747caef6f7 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/rep_barrier/9a266351ea0e4355896e1a747caef6f7 2023-04-25 09:56:45,749 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9a266351ea0e4355896e1a747caef6f7 2023-04-25 09:56:45,749 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/rep_barrier/9a266351ea0e4355896e1a747caef6f7, entries=6, sequenceid=93, filesize=5.5 K 2023-04-25 09:56:45,750 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/.tmp/table/3bb3be9613c24e3d821d22970a01e34e as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/table/3bb3be9613c24e3d821d22970a01e34e 2023-04-25 09:56:45,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3bb3be9613c24e3d821d22970a01e34e 2023-04-25 09:56:45,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/table/3bb3be9613c24e3d821d22970a01e34e, entries=9, sequenceid=93, filesize=5.5 K 2023-04-25 09:56:45,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 438ms, sequenceid=93, compaction requested=false 2023-04-25 09:56:45,757 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-25 09:56:45,758 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 09:56:45,759 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-25 09:56:45,759 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-25 09:56:45,759 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-25 09:56:45,889 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-25 09:56:45,889 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-25 09:56:45,916 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39407,1682416417123; all regions closed. 2023-04-25 09:56:45,923 DEBUG [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/oldWALs 2023-04-25 09:56:45,924 INFO [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39407%2C1682416417123.meta:.meta(num 1682416419208) 2023-04-25 09:56:45,928 DEBUG [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/oldWALs 2023-04-25 09:56:45,928 INFO [RS:0;jenkins-hbase4:39407] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39407%2C1682416417123:(num 1682416419029) 2023-04-25 09:56:45,928 DEBUG [RS:0;jenkins-hbase4:39407] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:45,928 INFO [RS:0;jenkins-hbase4:39407] regionserver.LeaseManager(133): Closed leases 2023-04-25 09:56:45,928 INFO [RS:0;jenkins-hbase4:39407] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-25 09:56:45,929 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-25 09:56:45,930 INFO [RS:0;jenkins-hbase4:39407] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39407 2023-04-25 09:56:45,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39407,1682416417123 2023-04-25 09:56:45,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 09:56:45,935 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-25 09:56:45,935 ERROR [Listener at localhost/33705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@39c58faf rejected from java.util.concurrent.ThreadPoolExecutor@3dfa0c4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2684] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 09:56:45,937 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39407,1682416417123] 2023-04-25 09:56:45,938 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39407,1682416417123; numProcessing=1 2023-04-25 09:56:45,939 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39407,1682416417123 already deleted, retry=false 2023-04-25 09:56:45,939 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39407,1682416417123 expired; onlineServers=0 2023-04-25 09:56:45,939 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33039,1682416415885' ***** 2023-04-25 09:56:45,939 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-25 09:56:45,939 DEBUG [M:0;jenkins-hbase4:33039] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@511e1b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-25 09:56:45,939 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33039,1682416415885 2023-04-25 09:56:45,939 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33039,1682416415885; all regions closed. 2023-04-25 09:56:45,939 DEBUG [M:0;jenkins-hbase4:33039] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-25 09:56:45,939 DEBUG [M:0;jenkins-hbase4:33039] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-25 09:56:45,939 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-25 09:56:45,939 DEBUG [M:0;jenkins-hbase4:33039] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-25 09:56:45,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682416418690] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682416418690,5,FailOnTimeoutGroup] 2023-04-25 09:56:45,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682416418687] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682416418687,5,FailOnTimeoutGroup] 2023-04-25 09:56:45,939 INFO [M:0;jenkins-hbase4:33039] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-25 09:56:45,940 INFO [M:0;jenkins-hbase4:33039] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-25 09:56:45,940 INFO [M:0;jenkins-hbase4:33039] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-25 09:56:45,940 DEBUG [M:0;jenkins-hbase4:33039] master.HMaster(1512): Stopping service threads 2023-04-25 09:56:45,940 INFO [M:0;jenkins-hbase4:33039] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-25 09:56:45,940 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-25 09:56:45,941 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-25 09:56:45,941 ERROR [M:0;jenkins-hbase4:33039] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-25 09:56:45,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-25 09:56:45,941 INFO [M:0;jenkins-hbase4:33039] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-25 09:56:45,941 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-25 09:56:45,942 DEBUG [M:0;jenkins-hbase4:33039] zookeeper.ZKUtil(398): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-25 09:56:45,942 WARN [M:0;jenkins-hbase4:33039] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-25 09:56:45,942 INFO [M:0;jenkins-hbase4:33039] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-25 09:56:45,942 INFO [M:0;jenkins-hbase4:33039] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-25 09:56:45,943 DEBUG [M:0;jenkins-hbase4:33039] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-25 09:56:45,943 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:56:45,943 DEBUG [M:0;jenkins-hbase4:33039] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:56:45,943 DEBUG [M:0;jenkins-hbase4:33039] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-25 09:56:45,943 DEBUG [M:0;jenkins-hbase4:33039] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:56:45,943 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.04 KB heapSize=393.73 KB 2023-04-25 09:56:45,951 INFO [M:0;jenkins-hbase4:33039] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.04 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/594413e34d804812ba7b4d31501cad68 2023-04-25 09:56:45,953 INFO [M:0;jenkins-hbase4:33039] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 594413e34d804812ba7b4d31501cad68 2023-04-25 09:56:45,954 DEBUG [M:0;jenkins-hbase4:33039] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/594413e34d804812ba7b4d31501cad68 as hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/594413e34d804812ba7b4d31501cad68 2023-04-25 09:56:45,957 INFO [M:0;jenkins-hbase4:33039] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 594413e34d804812ba7b4d31501cad68 2023-04-25 09:56:45,957 INFO [M:0;jenkins-hbase4:33039] regionserver.HStore(1080): Added hdfs://localhost:35439/user/jenkins/test-data/84c19924-b442-9ae1-fe62-e3a4dd68713c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/594413e34d804812ba7b4d31501cad68, entries=65, sequenceid=578, filesize=24.7 K 2023-04-25 09:56:45,957 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegion(2948): Finished flush of dataSize ~345.04 KB/353324, heapSize ~393.72 KB/403168, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 14ms, sequenceid=578, compaction requested=false 2023-04-25 09:56:45,958 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-25 09:56:45,958 DEBUG [M:0;jenkins-hbase4:33039] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-25 09:56:45,960 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-25 09:56:45,960 INFO [M:0;jenkins-hbase4:33039] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-25 09:56:45,961 INFO [M:0;jenkins-hbase4:33039] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33039 2023-04-25 09:56:45,963 DEBUG [M:0;jenkins-hbase4:33039] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,33039,1682416415885 already deleted, retry=false 2023-04-25 09:56:46,039 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 09:56:46,039 INFO [RS:0;jenkins-hbase4:39407] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39407,1682416417123; zookeeper connection closed. 2023-04-25 09:56:46,039 ERROR [Listener at localhost/33705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c672e2b rejected from java.util.concurrent.ThreadPoolExecutor@3dfa0c4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2684] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 09:56:46,039 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): regionserver:39407-0x101babd1af30001, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 09:56:46,039 ERROR [Listener at localhost/33705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24bf8ffd rejected from java.util.concurrent.ThreadPoolExecutor@3dfa0c4f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2684] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 09:56:46,039 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6bea5e04] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6bea5e04 2023-04-25 09:56:46,040 INFO [Listener at localhost/33705] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-25 09:56:46,052 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-25 09:56:46,138 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 09:56:46,138 INFO [M:0;jenkins-hbase4:33039] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33039,1682416415885; zookeeper connection closed. 2023-04-25 09:56:46,138 ERROR [Listener at localhost/33705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e57deaa rejected from java.util.concurrent.ThreadPoolExecutor@75a0b68f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4482] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 09:56:46,139 DEBUG [Listener at localhost/33705-EventThread] zookeeper.ZKWatcher(600): master:33039-0x101babd1af30000, quorum=127.0.0.1:60014, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-25 09:56:46,139 ERROR [Listener at localhost/33705-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2e523c7c rejected from java.util.concurrent.ThreadPoolExecutor@75a0b68f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4482] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-25 09:56:46,140 WARN [Listener at localhost/33705] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-25 09:56:46,143 INFO [Listener at localhost/33705] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-25 09:56:46,252 WARN [BP-1066761539-172.31.14.131-1682416412467 heartbeating to localhost/127.0.0.1:35439] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-25 09:56:46,252 WARN [BP-1066761539-172.31.14.131-1682416412467 heartbeating to localhost/127.0.0.1:35439] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1066761539-172.31.14.131-1682416412467 (Datanode Uuid 54203c7c-a588-4776-8ed2-7bd6392ca367) service to localhost/127.0.0.1:35439 2023-04-25 09:56:46,254 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/cluster_2925ad82-f88f-75b8-db74-5e3e3d9f1396/dfs/data/data1/current/BP-1066761539-172.31.14.131-1682416412467] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-25 09:56:46,255 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2a789c5e-d052-910d-404b-620d2b28ea0f/cluster_2925ad82-f88f-75b8-db74-5e3e3d9f1396/dfs/data/data2/current/BP-1066761539-172.31.14.131-1682416412467] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-25 09:56:46,288 INFO [Listener at localhost/33705] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-25 09:56:46,459 INFO [Listener at localhost/33705] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-25 09:56:46,639 INFO [Listener at localhost/33705] hbase.HBaseTestingUtility(1293): Minicluster is down